+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210518/A1e --output-run DEEP/20210518/A1e/science#step0/20240316T000255Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.4131 seconds; current memory usage: 0.178 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0037 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0050 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/pipeline_orig.qgraph DEEP/20210518/A1e/science#step0/20240316T000255Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.5110 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 6.5973 seconds; current memory usage: 0.270 Gibyte, delta: 0.092 Gibyte, peak delta: 0.080 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0099 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 186 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210518_A1e_science#step0_20240316T000255Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0479 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 188 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.1649 seconds; current memory usage: 0.272 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: c54c6b60-65ae-4b13-a17f-508788c56a1f parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-16 00:03:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-16 00:03:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 00:03:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-16 00:03:16 proc_lsst.multi:170 [INFO] [multi] job_id 13819 proc_lsst.multi INFO: [multi] job_id 13819 2024-03-16 00:03:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:16 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 13819 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 13819 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 13819 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash a66dc317568217f850e933e7659b7209 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470df4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b7762167-4775-4ac3-85a8-06b6dd7887b7_isrForCrosstalkSources_999140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b7762167-4775-4ac3-85a8-06b6dd7887b7_isrForCrosstalkSources_999140_44.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 154c11a7fee8d21872957797b6b7235a parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470df880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/201b96f8-1193-4c78-a58a-6cad074af834_isrForCrosstalkSources_999139_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/201b96f8-1193-4c78-a58a-6cad074af834_isrForCrosstalkSources_999139_54.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 19fa902d223af78b70008a601fe050ac parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470dfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a350945b-db69-4882-84b9-ec8ba298a1c9_isrForCrosstalkSources_999141_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a350945b-db69-4882-84b9-ec8ba298a1c9_isrForCrosstalkSources_999141_49.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 052623726f29d11276706feae4d5f94a parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470df9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/eb8625f4-240f-4ea7-9415-93eb5dc0be8f_isrForCrosstalkSources_999140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/eb8625f4-240f-4ea7-9415-93eb5dc0be8f_isrForCrosstalkSources_999140_31.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 9316651e21cc5f0a225e2ff126a3c9e9 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470dfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a035f6bd-91e3-475c-ac90-a693fe002efc_isrForCrosstalkSources_999141_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a035f6bd-91e3-475c-ac90-a693fe002efc_isrForCrosstalkSources_999141_26.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash e5d1d696c2d268efe04de3d435fd38d3 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470dfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/9441188f-d74c-4cf1-ba06-84cdbf874857_isrForCrosstalkSources_999141_28.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/9441188f-d74c-4cf1-ba06-84cdbf874857_isrForCrosstalkSources_999141_28.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash b9712403899ecafd869e9e7efeb6c829 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470df7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/0c5e44ec-889e-49a7-ae80-a7f15192f4cb_isrForCrosstalkSources_999141_16.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/0c5e44ec-889e-49a7-ae80-a7f15192f4cb_isrForCrosstalkSources_999141_16.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 128c78f1e8dade86390484b63a8d5786 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470dfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7456046a-0423-4d8e-881a-7d30f15d097b_isrForCrosstalkSources_999139_52.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7456046a-0423-4d8e-881a-7d30f15d097b_isrForCrosstalkSources_999139_52.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 114baf1a3f61021599edf3f8de1b8390 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470dede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/90a45290-a0d8-4f28-a08c-df587f1d7aa3_isrForCrosstalkSources_999141_10.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/90a45290-a0d8-4f28-a08c-df587f1d7aa3_isrForCrosstalkSources_999141_10.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 782721ab127e5b37a330867fe1b972ae parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470df6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4e0717d8-a768-4979-a991-e68b3b28422e_isrForCrosstalkSources_999139_28.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4e0717d8-a768-4979-a991-e68b3b28422e_isrForCrosstalkSources_999139_28.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 4a907880b0461b59543ba572a515091c parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470dfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1c4a5719-4c27-4e95-8fa1-548a15b1e55d_isrForCrosstalkSources_999139_30.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1c4a5719-4c27-4e95-8fa1-548a15b1e55d_isrForCrosstalkSources_999139_30.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash d5d89575edb6db8bb4953c9cd3132d78 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f470dfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/7f222673-a548-417e-8d60-f3b134b497ff_isrForCrosstalkSources_999141_31.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/7f222673-a548-417e-8d60-f3b134b497ff_isrForCrosstalkSources_999141_31.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash c8ea1ac2e28ba4fe508747879f47ebfb parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471200e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b0d4850f-2f24-49d1-9edc-729e93b38d5b_isrForCrosstalkSources_999140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b0d4850f-2f24-49d1-9edc-729e93b38d5b_isrForCrosstalkSources_999140_6.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 1dd43c4dbaebe25881210bc274e84a63 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/77dd27a8-d7d7-403b-b6a4-25509ce9c21a_isrForCrosstalkSources_999140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/77dd27a8-d7d7-403b-b6a4-25509ce9c21a_isrForCrosstalkSources_999140_8.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 77c304f9ed5e88e4dc992f6aca26b217 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471205e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2ac714de-3875-4d2f-a5e9-2fa45721448f_isrForCrosstalkSources_999141_39.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2ac714de-3875-4d2f-a5e9-2fa45721448f_isrForCrosstalkSources_999141_39.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 81562b5eb8484820d09d6feead50df48 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471202c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/460c7d69-6adb-484b-af05-a51323fec3a7_isrForCrosstalkSources_999141_45.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/460c7d69-6adb-484b-af05-a51323fec3a7_isrForCrosstalkSources_999141_45.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 628d045558474fb43df926210f37fc77 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a503c3a8-0156-4b3c-bc53-48994311200a_isrForCrosstalkSources_999139_7.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a503c3a8-0156-4b3c-bc53-48994311200a_isrForCrosstalkSources_999139_7.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash df072b6d59740f48866c521121a3e643 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cd555ae9-7142-4311-9d6c-57b71641955b_isrForCrosstalkSources_999140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cd555ae9-7142-4311-9d6c-57b71641955b_isrForCrosstalkSources_999140_19.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash a06d2859ad062e25357dfb4c03ad62d6 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/db532951-f2d3-4650-8118-3e8b8cd8ff63_isrForCrosstalkSources_999139_12.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/db532951-f2d3-4650-8118-3e8b8cd8ff63_isrForCrosstalkSources_999139_12.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash ef0eaf2816917579b0d2c9feee6e872b parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471207c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/6d3315fb-4017-41a3-b8b2-ffbf4e449ce5_isrForCrosstalkSources_999139_47.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/6d3315fb-4017-41a3-b8b2-ffbf4e449ce5_isrForCrosstalkSources_999139_47.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash c338e122551f314db37a984b60b7a467 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/d906cdcc-cb3f-4860-b6ee-08a52ab654e6_isrForCrosstalkSources_999139_2.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/d906cdcc-cb3f-4860-b6ee-08a52ab654e6_isrForCrosstalkSources_999139_2.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 639dd9f5504a4fb93d7cb025b9b65efa parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471209a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/ee470195-949b-4c54-a239-12fb17af9d46_isrForCrosstalkSources_999140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/ee470195-949b-4c54-a239-12fb17af9d46_isrForCrosstalkSources_999140_33.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 20369bfff92e50273a1c3496bc1b1e17 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/75dd38c7-ffdd-4515-9770-63a6fda67d63_isrForCrosstalkSources_999140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/75dd38c7-ffdd-4515-9770-63a6fda67d63_isrForCrosstalkSources_999140_23.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash ea8d29d59944d3a863a68e0477cb3079 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a776cc66-a856-479b-93cc-9f51378fa528_isrForCrosstalkSources_999140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a776cc66-a856-479b-93cc-9f51378fa528_isrForCrosstalkSources_999140_4.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash cbe95063ee0dce3277f54dcd7f5f06bc parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/905633c6-4b15-4341-9a25-2f35d34c5204_isrForCrosstalkSources_999140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/905633c6-4b15-4341-9a25-2f35d34c5204_isrForCrosstalkSources_999140_17.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 5da0aa29cfdb7b7422fa03443a86eb26 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/11d8deba-80d9-41d3-ad2f-9b002fefff94_isrForCrosstalkSources_999140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/11d8deba-80d9-41d3-ad2f-9b002fefff94_isrForCrosstalkSources_999140_13.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 8c2013d9011ab8747791d5ecfba25cad parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a2681153-7484-4b9c-805c-cef18a608d2c_isrForCrosstalkSources_999140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a2681153-7484-4b9c-805c-cef18a608d2c_isrForCrosstalkSources_999140_20.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash a35733f86f0780d9338f8deaa1a1a6e7 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/75f5d7fc-9e1e-4bf4-a5ce-0aefd16fac76_isrForCrosstalkSources_999141_60.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/75f5d7fc-9e1e-4bf4-a5ce-0aefd16fac76_isrForCrosstalkSources_999141_60.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 0fa1c37e87d43fd4aa0b20c0540c2e15 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4db68bae-ab93-432c-9f6c-e01ba03d26c3_isrForCrosstalkSources_999140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4db68bae-ab93-432c-9f6c-e01ba03d26c3_isrForCrosstalkSources_999140_29.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash d427b2c414162afcddb0ec336f33cf1c parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/fa765250-4956-480f-aecb-5e3a9f88ad8b_isrForCrosstalkSources_999139_34.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/fa765250-4956-480f-aecb-5e3a9f88ad8b_isrForCrosstalkSources_999139_34.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 2df474d3629284a7ae49f53284f5690d parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/721285ea-61ad-4e7b-bf6a-98510cbcce53_isrForCrosstalkSources_999139_4.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/721285ea-61ad-4e7b-bf6a-98510cbcce53_isrForCrosstalkSources_999139_4.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 012e7dd1204dd70c69d69c0b9fd274f0 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/150c83a2-bd37-4e85-bfa4-17e4df157c1b_isrForCrosstalkSources_999139_19.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/150c83a2-bd37-4e85-bfa4-17e4df157c1b_isrForCrosstalkSources_999139_19.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 85a3972e566a2b68b92a527782766314 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/745f2920-5d70-4e60-a815-a095c9a20138_isrForCrosstalkSources_999139_50.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/745f2920-5d70-4e60-a815-a095c9a20138_isrForCrosstalkSources_999139_50.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 2b5e8db95a329f721d18243cae3db9ed parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/950bfe16-8f0c-49c1-a803-dc7f462daa29_isrForCrosstalkSources_999141_8.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/950bfe16-8f0c-49c1-a803-dc7f462daa29_isrForCrosstalkSources_999141_8.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 9708fb2b930864f0ebb481f64ff1be5e parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a4f15022-2020-40ad-918d-88a7d87b48c8_isrForCrosstalkSources_999139_31.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a4f15022-2020-40ad-918d-88a7d87b48c8_isrForCrosstalkSources_999139_31.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash a44a7942e48bb11e590540b3804d6330 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c10aa9b2-3aa0-44af-9e65-56766b06c4b3_isrForCrosstalkSources_999140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c10aa9b2-3aa0-44af-9e65-56766b06c4b3_isrForCrosstalkSources_999140_48.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 93b4048d20528c9782a7dd43fc0c49ef parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/f3939eb8-9910-40be-a91f-3547c37c1c30_isrForCrosstalkSources_999139_15.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/f3939eb8-9910-40be-a91f-3547c37c1c30_isrForCrosstalkSources_999139_15.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 89a0739db896462509efc3e627a75e01 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/be36e612-b292-4314-a8b4-a81e2999a78f_isrForCrosstalkSources_999141_36.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/be36e612-b292-4314-a8b4-a81e2999a78f_isrForCrosstalkSources_999141_36.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 7eb1bf2836333978209bd33f9f944655 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9d2776ef-62eb-410f-ae9e-e2f60e230dc0_isrForCrosstalkSources_999140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9d2776ef-62eb-410f-ae9e-e2f60e230dc0_isrForCrosstalkSources_999140_21.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 3784491c05ea3e5b799bd2398443099e parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/32cbe50e-f9e3-41f8-a88f-4d047bfe5a95_isrForCrosstalkSources_999140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/32cbe50e-f9e3-41f8-a88f-4d047bfe5a95_isrForCrosstalkSources_999140_16.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 5c37b094ad8701abb820f3e9bba63acb parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471214e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/54e62f14-4873-4a0e-80e7-ceeba6a615e0_isrForCrosstalkSources_999140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/54e62f14-4873-4a0e-80e7-ceeba6a615e0_isrForCrosstalkSources_999140_41.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 87da6b57e0640e7b6430d6967a213612 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/f939bb27-0bd4-4f57-b9cf-38fdf6718d6e_isrForCrosstalkSources_999139_49.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/f939bb27-0bd4-4f57-b9cf-38fdf6718d6e_isrForCrosstalkSources_999139_49.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 679c261fab2ecf033bda101bf17d73b3 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/480f22d6-639f-472a-a3fc-47b9c9c69304_isrForCrosstalkSources_999139_14.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/480f22d6-639f-472a-a3fc-47b9c9c69304_isrForCrosstalkSources_999139_14.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 7f32b85d1f8325d2fb9b570c8d5c18f3 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471216c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/ba8b528d-4e69-46fd-a155-392181459c50_isrForCrosstalkSources_999139_9.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/ba8b528d-4e69-46fd-a155-392181459c50_isrForCrosstalkSources_999139_9.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 7434ba3afa612dffc8927cc642ce4d66 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5e026b0d-e61b-41ce-9c7c-201a79ae73e0_isrForCrosstalkSources_999139_20.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5e026b0d-e61b-41ce-9c7c-201a79ae73e0_isrForCrosstalkSources_999139_20.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 023a7d927ddf9518b7b4a3780ca9260d parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/90d99ddf-79b0-4537-9cd2-5a5a500e4e8c_isrForCrosstalkSources_999141_20.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/90d99ddf-79b0-4537-9cd2-5a5a500e4e8c_isrForCrosstalkSources_999141_20.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 4ec325b48d240a1181d6334125404f52 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/adf1ce92-5d02-4333-a679-d906bf85d8a7_isrForCrosstalkSources_999139_62.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/adf1ce92-5d02-4333-a679-d906bf85d8a7_isrForCrosstalkSources_999139_62.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 5073575a66bab2100ecf154daf345815 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471218a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/245e27df-37cc-4816-b7d8-c9174b3c2fb2_isrForCrosstalkSources_999139_45.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/245e27df-37cc-4816-b7d8-c9174b3c2fb2_isrForCrosstalkSources_999139_45.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 52bf9a7f3b0519c2ae3b05560baea28f parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471219e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e9ecf5c5-3bf5-4fce-8583-fd8e9790e3c4_isrForCrosstalkSources_999141_24.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e9ecf5c5-3bf5-4fce-8583-fd8e9790e3c4_isrForCrosstalkSources_999141_24.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 0913067b49a39ab4e6674e0130b3c1de parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/28dac436-8ec7-454d-8f7a-7f1f236ad1fd_isrForCrosstalkSources_999140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/28dac436-8ec7-454d-8f7a-7f1f236ad1fd_isrForCrosstalkSources_999140_40.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 19cd730ea8161a08a3bdf33bbb9c28d7 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4fd12186-1f06-496c-b2dd-ba85b31b78fb_isrForCrosstalkSources_999140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4fd12186-1f06-496c-b2dd-ba85b31b78fb_isrForCrosstalkSources_999140_30.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash f8cbcded28aea598261742fbc868d736 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/f167852d-1030-47c0-b471-b16bbeecdc37_isrForCrosstalkSources_999140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/f167852d-1030-47c0-b471-b16bbeecdc37_isrForCrosstalkSources_999140_59.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 2ef64a81d3ed7f5664909fb0304579d0 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/045c4ef9-44ec-4610-ba4a-7c01d1a721f2_isrForCrosstalkSources_999140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/045c4ef9-44ec-4610-ba4a-7c01d1a721f2_isrForCrosstalkSources_999140_25.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 3b1cbd828344006390d0932067a6ef66 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ee8e4df8-a099-4375-9b6f-fb6ed0559c90_isrForCrosstalkSources_999141_47.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ee8e4df8-a099-4375-9b6f-fb6ed0559c90_isrForCrosstalkSources_999141_47.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash ef4e253197c5652b073c0315412b92bb parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/d4b48fe2-c9d0-4096-a267-615b3c9b498f_isrForCrosstalkSources_999140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/d4b48fe2-c9d0-4096-a267-615b3c9b498f_isrForCrosstalkSources_999140_28.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 1482628fb4d9269148a9984f5f851b09 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/0d192347-dc69-4c5a-b45e-9ff6d498d1a2_isrForCrosstalkSources_999139_25.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/0d192347-dc69-4c5a-b45e-9ff6d498d1a2_isrForCrosstalkSources_999139_25.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 55a3862630ea922d5f3e9aa1e6b71d5b parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/daf44ad0-02ff-45be-b32f-0cfde0aa424f_isrForCrosstalkSources_999139_61.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/daf44ad0-02ff-45be-b32f-0cfde0aa424f_isrForCrosstalkSources_999139_61.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 86244beff8ed745c561c9c27daecc605 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47121ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/0284ffe9-97f6-4780-8614-83234540c87f_isrForCrosstalkSources_999141_38.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/0284ffe9-97f6-4780-8614-83234540c87f_isrForCrosstalkSources_999141_38.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 04ef39f11f6e7a12ffe46c169d2ce421 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/c33fd977-ceb9-4df3-ad0b-011c3ce6acbb_isrForCrosstalkSources_999139_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/c33fd977-ceb9-4df3-ad0b-011c3ce6acbb_isrForCrosstalkSources_999139_18.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 6a2c7a6a621cdb0645878e21b33d1dc0 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/d9db02b1-6d3f-42fb-a779-47512785f27b_isrForCrosstalkSources_999139_24.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/d9db02b1-6d3f-42fb-a779-47512785f27b_isrForCrosstalkSources_999139_24.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 763cb84ae48968eabdf969930ac69e09 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/20f73278-e429-4168-b1a9-ef9f4e24b541_isrForCrosstalkSources_999140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/20f73278-e429-4168-b1a9-ef9f4e24b541_isrForCrosstalkSources_999140_55.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash e1c2f1f2941aed8569de398a26a77c3c parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/14826200-eb2f-40ac-a53a-c46dcc8c9d27_isrForCrosstalkSources_999141_15.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/14826200-eb2f-40ac-a53a-c46dcc8c9d27_isrForCrosstalkSources_999141_15.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 0caf446dd7474e1c3dfc51d182188781 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/384b6f44-b6fa-4b2b-98c5-535918ee4d2f_isrForCrosstalkSources_999140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/384b6f44-b6fa-4b2b-98c5-535918ee4d2f_isrForCrosstalkSources_999140_60.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 9ee446058085d66b513f7662683d9d97 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1e8ac794-9eaf-4d45-80ab-218256491469_isrForCrosstalkSources_999139_37.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1e8ac794-9eaf-4d45-80ab-218256491469_isrForCrosstalkSources_999139_37.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash d9f1348748a3ac6eeb1b490daaf33943 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471223e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/8a3275f3-0062-47b5-846a-f46d5c92e8d2_isrForCrosstalkSources_999140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/8a3275f3-0062-47b5-846a-f46d5c92e8d2_isrForCrosstalkSources_999140_45.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 92ef788e50cf2811ab8589a0cb0af029 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/8a2f3425-8fb6-437b-9cce-58f5cb950aa5_isrForCrosstalkSources_999139_40.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/8a2f3425-8fb6-437b-9cce-58f5cb950aa5_isrForCrosstalkSources_999139_40.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash a231ac167d7c14edc79ce348316963b4 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471225c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/111470a8-9adb-47c9-8d50-a0d7bf99ce56_isrForCrosstalkSources_999139_43.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/111470a8-9adb-47c9-8d50-a0d7bf99ce56_isrForCrosstalkSources_999139_43.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 799a9b719fae0bba497691e225963a46 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/6936e32a-d665-41b3-9b35-1f22338cfaf0_isrForCrosstalkSources_999141_12.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/6936e32a-d665-41b3-9b35-1f22338cfaf0_isrForCrosstalkSources_999141_12.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 92aa8252c70e2ea48781021aca5d616b parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/099a7248-1d3b-46c2-965a-ca2e6d6c9c9a_isrForCrosstalkSources_999141_9.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/099a7248-1d3b-46c2-965a-ca2e6d6c9c9a_isrForCrosstalkSources_999141_9.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash a6f3ed5829569eda8901ea2ef2d30301 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a3c2c2f5-67ea-48c2-8520-b55ec3a8e875_isrForCrosstalkSources_999140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a3c2c2f5-67ea-48c2-8520-b55ec3a8e875_isrForCrosstalkSources_999140_56.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 5103ed2e71163d5cbe772e38904ea732 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471227a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cce00877-e37b-4d94-baf6-b1ae9deac7da_isrForCrosstalkSources_999140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cce00877-e37b-4d94-baf6-b1ae9deac7da_isrForCrosstalkSources_999140_14.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash c8deba3271f82ee487a7760cf894de3f parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471228e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/1c298443-a8c5-4579-8ef1-6c482b5ff0de_isrForCrosstalkSources_999141_2.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/1c298443-a8c5-4579-8ef1-6c482b5ff0de_isrForCrosstalkSources_999141_2.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 6dba80cec8af6ec1bda4f61efba3105f parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/aa7248bd-03fa-478f-8c25-ca10375ac813_isrForCrosstalkSources_999141_57.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/aa7248bd-03fa-478f-8c25-ca10375ac813_isrForCrosstalkSources_999141_57.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 7997d22a5ba0cb94f3c58fb69a1dce2d parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2b6986f2-28ce-4441-bdf9-5a6739049ea1_isrForCrosstalkSources_999141_56.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2b6986f2-28ce-4441-bdf9-5a6739049ea1_isrForCrosstalkSources_999141_56.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 9e0198462da567f9916f05c82d316d4e parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/c0419f8e-1873-4513-ae81-a71a55d839ec_isrForCrosstalkSources_999141_27.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/c0419f8e-1873-4513-ae81-a71a55d839ec_isrForCrosstalkSources_999141_27.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 2d0eac13f54d5eaf7de40ef875fe4792 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/19796626-63fb-4d7e-a26b-58ca5791e2c4_isrForCrosstalkSources_999140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/19796626-63fb-4d7e-a26b-58ca5791e2c4_isrForCrosstalkSources_999140_2.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash cef2afe7ba66dae9ba0c0773523239f2 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5107bb27-5471-4e69-bb0d-c1124631a95d_isrForCrosstalkSources_999139_42.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5107bb27-5471-4e69-bb0d-c1124631a95d_isrForCrosstalkSources_999139_42.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash c7b3672fedcb731a7cabef3acb255270 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/faf637b9-7e9a-4d02-b652-5f3b152b026d_isrForCrosstalkSources_999140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/faf637b9-7e9a-4d02-b652-5f3b152b026d_isrForCrosstalkSources_999140_39.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 22cf7f7e705031bb06e98cd82f6b24ee parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9d91fe40-fc9b-4d7f-9c2f-c9e27d5e38a3_isrForCrosstalkSources_999140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9d91fe40-fc9b-4d7f-9c2f-c9e27d5e38a3_isrForCrosstalkSources_999140_54.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 2cb92b3fd2aea6a11b37f7e5a7033086 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/468eda4d-f8ec-439f-a43f-15dc834c4dba_isrForCrosstalkSources_999141_1.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/468eda4d-f8ec-439f-a43f-15dc834c4dba_isrForCrosstalkSources_999141_1.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 0171fb8c3b9d559c093eeb03d3130813 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/75f5c6d5-83b7-452c-a479-fbd129044b45_isrForCrosstalkSources_999140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/75f5c6d5-83b7-452c-a479-fbd129044b45_isrForCrosstalkSources_999140_46.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 0762d2219bf87677f3c83bb3a3a2a480 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c0480bc6-a5ed-40a5-9081-410247382973_isrForCrosstalkSources_999140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c0480bc6-a5ed-40a5-9081-410247382973_isrForCrosstalkSources_999140_50.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 7a59aef93645326808b0f5cea8ec5fb8 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/8adaa531-b601-4194-b0af-2630b26fce08_isrForCrosstalkSources_999139_22.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/8adaa531-b601-4194-b0af-2630b26fce08_isrForCrosstalkSources_999139_22.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash df3d3725bc6393844604fee67cf3829d parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47122fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f9c2637a-08d5-48f5-824f-100e55eec0e9_isrForCrosstalkSources_999141_22.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f9c2637a-08d5-48f5-824f-100e55eec0e9_isrForCrosstalkSources_999141_22.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 2011039bafbaa3c4bc3e9d9674f38762 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7b15f7c6-877f-4b3d-a841-84528bc65cc8_isrForCrosstalkSources_999139_36.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7b15f7c6-877f-4b3d-a841-84528bc65cc8_isrForCrosstalkSources_999139_36.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash f76ad3c18714ec8894b657324521bc47 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/313e6afa-dd1a-4fde-8b0b-27a69d4f6bd6_isrForCrosstalkSources_999139_35.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/313e6afa-dd1a-4fde-8b0b-27a69d4f6bd6_isrForCrosstalkSources_999139_35.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 1665f6af42f2606383bf352ecc53ef0c parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1a2c6e5f-fd59-4b4e-b589-ba6ae73a4c5a_isrForCrosstalkSources_999139_29.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1a2c6e5f-fd59-4b4e-b589-ba6ae73a4c5a_isrForCrosstalkSources_999139_29.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash cfbc530a4996181e10e03d3f411d25c4 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a179b5ca-7897-4708-8086-b50ddc84466e_isrForCrosstalkSources_999139_23.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a179b5ca-7897-4708-8086-b50ddc84466e_isrForCrosstalkSources_999139_23.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 11a31073eaf88a073b69eca267ced1a4 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471232e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/caa78fa7-cef6-4981-a70e-d8c934c55a9d_isrForCrosstalkSources_999140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/caa78fa7-cef6-4981-a70e-d8c934c55a9d_isrForCrosstalkSources_999140_62.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 4e9e2cc68d6e0a05020c2d674c9bc184 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e5cb1055-26c1-41c8-9b7b-c912d143045e_isrForCrosstalkSources_999141_50.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e5cb1055-26c1-41c8-9b7b-c912d143045e_isrForCrosstalkSources_999141_50.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 79c9af32f0b1c37166c9f9640390bfcb parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fdc31b65-30ae-468a-9520-9b35e1bece48_isrForCrosstalkSources_999141_37.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fdc31b65-30ae-468a-9520-9b35e1bece48_isrForCrosstalkSources_999141_37.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 96d885ae5ddd25130604512bfc7e4314 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/23868abd-7724-426b-8901-22f21662897f_isrForCrosstalkSources_999139_1.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/23868abd-7724-426b-8901-22f21662897f_isrForCrosstalkSources_999139_1.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash f5d94d3806e93e0785fa4580b1ecb9c0 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471234c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a70615e1-cb36-40fa-b0b0-3a648e3bf6e8_isrForCrosstalkSources_999139_33.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a70615e1-cb36-40fa-b0b0-3a648e3bf6e8_isrForCrosstalkSources_999139_33.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 12ac979947d60eaa7e805d3ec3f655e7 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/d4678a23-0282-44d7-8274-946b5465fdf7_isrForCrosstalkSources_999141_18.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/d4678a23-0282-44d7-8274-946b5465fdf7_isrForCrosstalkSources_999141_18.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 80aff07c5f557b590deb1a2cebfbf8ea parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471236a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/3a894255-6a1e-4e3a-9bcc-85eb73d20104_isrForCrosstalkSources_999140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/3a894255-6a1e-4e3a-9bcc-85eb73d20104_isrForCrosstalkSources_999140_3.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 8dc805ddc65ba0e60024c2b730b8829e parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/562a6acf-f269-45cf-9d7b-f674f3b2d458_isrForCrosstalkSources_999139_53.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/562a6acf-f269-45cf-9d7b-f674f3b2d458_isrForCrosstalkSources_999139_53.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash df11abd297ea0f3aceab54e6e328792a parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/01d301ed-813c-40a0-ada3-975317ac83eb_isrForCrosstalkSources_999139_57.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/01d301ed-813c-40a0-ada3-975317ac83eb_isrForCrosstalkSources_999139_57.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 9db8e43975f50913e4d4e5a566b34639 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471237e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/50f74c17-75f0-4c90-b327-b9435713d65a_isrForCrosstalkSources_999139_21.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/50f74c17-75f0-4c90-b327-b9435713d65a_isrForCrosstalkSources_999139_21.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash c01f1a34695980d89dbe54120c5f9cbc parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/824e063e-5d01-43aa-9963-907f13244389_isrForCrosstalkSources_999141_48.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/824e063e-5d01-43aa-9963-907f13244389_isrForCrosstalkSources_999141_48.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 32c88a14ce362981847e0bc48bfe0a34 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f471239c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/bcbe899b-1b01-4b01-bd25-f501540a7ac2_isrForCrosstalkSources_999140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/bcbe899b-1b01-4b01-bd25-f501540a7ac2_isrForCrosstalkSources_999140_10.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 426f6d8cad8d5ddb90ef305271acc0ae parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cc4d1950-dabe-425b-88ad-a95a35dde9f5_isrForCrosstalkSources_999140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cc4d1950-dabe-425b-88ad-a95a35dde9f5_isrForCrosstalkSources_999140_27.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 66874716acdf67adf6f581d464d51869 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/460a9e9d-845a-400d-840f-da1a00c50888_isrForCrosstalkSources_999141_35.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/460a9e9d-845a-400d-840f-da1a00c50888_isrForCrosstalkSources_999141_35.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 0b61e0f6efd4fe536ed4e39563c39e59 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cd1413d7-efcd-47da-9fd2-860dbb539dfd_isrForCrosstalkSources_999140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cd1413d7-efcd-47da-9fd2-860dbb539dfd_isrForCrosstalkSources_999140_26.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash bdd6e4470f6394967f64fbf137ec4b39 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/afebc454-4b3e-4a42-97dc-f5686de856c7_isrForCrosstalkSources_999141_34.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/afebc454-4b3e-4a42-97dc-f5686de856c7_isrForCrosstalkSources_999141_34.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash a387dcd44e5cb4ce6f27fd276419158c parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2eef36bc-3953-4229-9a3e-4de3c43a5fb3_isrForCrosstalkSources_999141_41.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2eef36bc-3953-4229-9a3e-4de3c43a5fb3_isrForCrosstalkSources_999141_41.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash decba366c7dab214a109fb5de7646172 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5d813990-1b33-424b-ac28-ed8de4059297_isrForCrosstalkSources_999139_10.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5d813990-1b33-424b-ac28-ed8de4059297_isrForCrosstalkSources_999139_10.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 021d0cea6258cdd23c18eb0873357d88 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/b9a4cd01-62c4-48f5-a394-f1604031d935_isrForCrosstalkSources_999139_16.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/b9a4cd01-62c4-48f5-a394-f1604031d935_isrForCrosstalkSources_999139_16.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash ac9f715799d3e6ca962189762bfd5805 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/987d404d-ed01-41f0-b48a-11696c787166_isrForCrosstalkSources_999139_5.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/987d404d-ed01-41f0-b48a-11696c787166_isrForCrosstalkSources_999139_5.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 9b820018cf96a8761429ae7f3fd1377d parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47120680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/d8535222-069c-482e-b19c-6f3de79a4567_isrForCrosstalkSources_999141_14.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/d8535222-069c-482e-b19c-6f3de79a4567_isrForCrosstalkSources_999141_14.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 47ac259b73926277a534e9a5d00c9a6c parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/24926554-8d84-4b79-80fb-6f5d85cd1f3e_isrForCrosstalkSources_999141_51.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/24926554-8d84-4b79-80fb-6f5d85cd1f3e_isrForCrosstalkSources_999141_51.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash d406c9d7413570d4815418e5776c4fd1 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/36a114e8-bf5e-4e36-bef1-8059285e5751_isrForCrosstalkSources_999141_23.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/36a114e8-bf5e-4e36-bef1-8059285e5751_isrForCrosstalkSources_999141_23.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash e3ad131548f46151ce6f12b464bb4174 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/4a995351-da31-473f-99ee-2e4409780701_isrForCrosstalkSources_999141_13.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/4a995351-da31-473f-99ee-2e4409780701_isrForCrosstalkSources_999141_13.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 7f684d94e9c667a11e22132127ccbd48 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9bd3b7b4-1e08-442b-b3a6-c9c1aedb2993_isrForCrosstalkSources_999140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9bd3b7b4-1e08-442b-b3a6-c9c1aedb2993_isrForCrosstalkSources_999140_57.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash cbf4172ef2cc9dfbc50cf44e10e17d75 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/e557be45-33d1-4f2a-bb84-4533ffb5fd8b_isrForCrosstalkSources_999139_6.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/e557be45-33d1-4f2a-bb84-4533ffb5fd8b_isrForCrosstalkSources_999139_6.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash bc8245ce2a22996fc7f511f09fa2dfa0 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/688ca4cd-eee0-47c6-a919-948658526ecb_isrForCrosstalkSources_999139_56.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/688ca4cd-eee0-47c6-a919-948658526ecb_isrForCrosstalkSources_999139_56.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash c73f747e3a215f2dc9b49e22c1fee8c0 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b9330f32-c7e5-4b05-be46-3b36e18fe6be_isrForCrosstalkSources_999140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b9330f32-c7e5-4b05-be46-3b36e18fe6be_isrForCrosstalkSources_999140_1.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 6f09c91c40fb7351af9323da669978bf parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e1d940c3-f11b-4eda-8158-778200594811_isrForCrosstalkSources_999140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e1d940c3-f11b-4eda-8158-778200594811_isrForCrosstalkSources_999140_24.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 3a83a1efa26900c252eac33e150d3d2d parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b745e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/05e0841c-9d45-424c-82e7-88b982bfcebd_isrForCrosstalkSources_999139_27.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/05e0841c-9d45-424c-82e7-88b982bfcebd_isrForCrosstalkSources_999139_27.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 354f1d1bae18875a489a78c59a5cbc53 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/fe19ee28-829e-4b91-af0d-0cb40330745d_isrForCrosstalkSources_999139_39.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/fe19ee28-829e-4b91-af0d-0cb40330745d_isrForCrosstalkSources_999139_39.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 3f46a8dd91c3943dd7c498b72958d0d7 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ef69143e-0545-4c04-85f7-75c2447ba6ca_isrForCrosstalkSources_999141_62.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ef69143e-0545-4c04-85f7-75c2447ba6ca_isrForCrosstalkSources_999141_62.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 7e1dddef279d82f9692fa66617ba5740 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/8916f19b-9f60-42c1-983b-25eb8e23a0fd_isrForCrosstalkSources_999141_29.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/8916f19b-9f60-42c1-983b-25eb8e23a0fd_isrForCrosstalkSources_999141_29.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 50ae189c1329dd5d016ae19c552f0b78 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4761dde5-5f5a-4886-8172-12f525336ae9_isrForCrosstalkSources_999139_32.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4761dde5-5f5a-4886-8172-12f525336ae9_isrForCrosstalkSources_999139_32.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash f6d542995f6c4990e58ff15ba228c04b parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/70be6fe1-8a45-4eff-a076-524122c8771a_isrForCrosstalkSources_999141_61.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/70be6fe1-8a45-4eff-a076-524122c8771a_isrForCrosstalkSources_999141_61.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 40f12bd6c18945bcd3b0fe706994f124 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/854e2e01-61c7-4efd-94c4-eb8eed30b932_isrForCrosstalkSources_999141_54.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/854e2e01-61c7-4efd-94c4-eb8eed30b932_isrForCrosstalkSources_999141_54.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 0d2b97a3ca5ae5dcd837058486108a18 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/4eb81f9a-9436-461d-9f8a-38eacfeb0443_isrForCrosstalkSources_999141_19.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/4eb81f9a-9436-461d-9f8a-38eacfeb0443_isrForCrosstalkSources_999141_19.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 829df48e864e9c3c4aceea5688ff0eaa parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/49ef8480-35f3-4fc0-b9f2-af507ccfdb2c_isrForCrosstalkSources_999141_21.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/49ef8480-35f3-4fc0-b9f2-af507ccfdb2c_isrForCrosstalkSources_999141_21.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash ab2078876788947a18166010e6146c28 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/0540d748-cd91-4a3a-a4ae-bca5b0e3e770_isrForCrosstalkSources_999140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/0540d748-cd91-4a3a-a4ae-bca5b0e3e770_isrForCrosstalkSources_999140_5.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash e80373c163c36746e3162a0a57eb0c16 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/eb517a6a-ee0c-4666-892f-30e88be51ddb_isrForCrosstalkSources_999139_17.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/eb517a6a-ee0c-4666-892f-30e88be51ddb_isrForCrosstalkSources_999139_17.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash ddbdd8f367f6b0a34bff2cbfceaa3c48 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a6586ecf-be9a-4d1b-b3b2-dc1dbe9adc47_isrForCrosstalkSources_999141_4.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a6586ecf-be9a-4d1b-b3b2-dc1dbe9adc47_isrForCrosstalkSources_999141_4.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 16ac508832d7268244ebde3662a6df70 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cffd2098-0880-45fc-a25f-972605e94f36_isrForCrosstalkSources_999140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cffd2098-0880-45fc-a25f-972605e94f36_isrForCrosstalkSources_999140_15.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash b746ab3b54efc89bbc075e98d513a13a parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/ff2281fa-489e-45e1-9d9c-b43db3718203_isrForCrosstalkSources_999140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/ff2281fa-489e-45e1-9d9c-b43db3718203_isrForCrosstalkSources_999140_32.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 7a10d4d57e2957e533ee2cf2062e0013 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/b310f608-77d5-4bbf-ad1a-8bb586e52b25_isrForCrosstalkSources_999141_55.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/b310f608-77d5-4bbf-ad1a-8bb586e52b25_isrForCrosstalkSources_999141_55.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash cf718e5215c49ceed384a1333004de8f parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/b6dff8a5-3177-4103-8ec0-f78da015f308_isrForCrosstalkSources_999139_38.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/b6dff8a5-3177-4103-8ec0-f78da015f308_isrForCrosstalkSources_999139_38.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 84a63994bfc550c1e178493b016dd792 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b74fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/5f8806ce-e91f-4e6d-b599-720290235fd3_isrForCrosstalkSources_999140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/5f8806ce-e91f-4e6d-b599-720290235fd3_isrForCrosstalkSources_999140_51.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash c586f1b5cbd877506bcc6dbf935707ee parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/710c5390-5648-40de-9c92-5e90dab29c4d_isrForCrosstalkSources_999139_48.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/710c5390-5648-40de-9c92-5e90dab29c4d_isrForCrosstalkSources_999139_48.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 14ca08c52a16871172c887fc2b25eabd parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4865bd85-a860-4e8a-8242-adb3446a68fa_isrForCrosstalkSources_999140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4865bd85-a860-4e8a-8242-adb3446a68fa_isrForCrosstalkSources_999140_49.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 7592b2d35fe1e7ee9837228585505de6 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/159f3402-4837-4527-a533-a3964ad9f61f_isrForCrosstalkSources_999139_44.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/159f3402-4837-4527-a533-a3964ad9f61f_isrForCrosstalkSources_999139_44.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash f493cb85cffedbef480ecef37f3b280a parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/3874738b-22e7-42b8-a186-e9d1e96c3595_isrForCrosstalkSources_999141_3.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/3874738b-22e7-42b8-a186-e9d1e96c3595_isrForCrosstalkSources_999141_3.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 61d53507b3735d8ef1a3352b8ad57b3b parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/1fdfba6f-a689-4c09-95e0-d4c2c2e77916_isrForCrosstalkSources_999141_40.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/1fdfba6f-a689-4c09-95e0-d4c2c2e77916_isrForCrosstalkSources_999141_40.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 92084fea8f88d97e8a75c0ad6cbd8c20 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/da192a09-d482-406e-aae0-b3c1931b7bbf_isrForCrosstalkSources_999139_51.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/da192a09-d482-406e-aae0-b3c1931b7bbf_isrForCrosstalkSources_999139_51.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 7732345aa46626fe2335c90fb6873ab0 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/43a4ab38-25f0-4a7f-874e-9263f0bc3960_isrForCrosstalkSources_999141_7.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/43a4ab38-25f0-4a7f-874e-9263f0bc3960_isrForCrosstalkSources_999141_7.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 433e38f1c297f045fe1be9800ef6c331 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b6b066f1-08c1-4f16-999e-2bf8f20aa490_isrForCrosstalkSources_999140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b6b066f1-08c1-4f16-999e-2bf8f20aa490_isrForCrosstalkSources_999140_7.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 4cf8b62452f9ba55ceef17542477f356 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a90c719a-4a51-49b1-9bed-f7359d7f71bf_isrForCrosstalkSources_999140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a90c719a-4a51-49b1-9bed-f7359d7f71bf_isrForCrosstalkSources_999140_18.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 9a80fe4bf04e8b8c5f8e5b0e866c8dc6 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/847e8b4a-19ab-44dc-811f-6a0adeef9877_isrForCrosstalkSources_999139_41.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/847e8b4a-19ab-44dc-811f-6a0adeef9877_isrForCrosstalkSources_999139_41.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash adb27294db730505bf9d1f3bdb8872d3 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b756c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7d727e03-6c7e-4d0a-bddf-2033eab7e70b_isrForCrosstalkSources_999139_46.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7d727e03-6c7e-4d0a-bddf-2033eab7e70b_isrForCrosstalkSources_999139_46.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 88d4e8f90512810e0879f0f4b601a38a parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e1f4f340-b147-4d71-9b34-a761f79377bd_isrForCrosstalkSources_999140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e1f4f340-b147-4d71-9b34-a761f79377bd_isrForCrosstalkSources_999140_35.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash f4730828a9aca0f6ecadca46541a390c parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/0b61a90d-c04f-414e-8481-506099ae5f0f_isrForCrosstalkSources_999140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/0b61a90d-c04f-414e-8481-506099ae5f0f_isrForCrosstalkSources_999140_37.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 9f805ae214e249449eb8abee0b4d3b00 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/bc9a4eb6-26fb-4ab1-9c93-a1717e4c65cf_isrForCrosstalkSources_999139_59.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/bc9a4eb6-26fb-4ab1-9c93-a1717e4c65cf_isrForCrosstalkSources_999139_59.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 0c4854461261c241956439598f06890e parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fd88e0cd-d10c-41c8-a818-100f7b69d508_isrForCrosstalkSources_999141_33.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fd88e0cd-d10c-41c8-a818-100f7b69d508_isrForCrosstalkSources_999141_33.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout 2024-03-16 00:03:17 proc_lsst.multi:146 [INFO] found job 13819 in provider local proc_lsst.multi INFO: found job 13819 in provider local parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 2485a4df25651824b231074822c3437c parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/0cfc68a0-f7c3-404a-bd55-e0e860d26938_isrForCrosstalkSources_999139_3.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/0cfc68a0-f7c3-404a-bd55-e0e860d26938_isrForCrosstalkSources_999139_3.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 1a6e405e9ecbb6fecdc27b53fcac229b parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b77f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/5c77d690-6cb9-4fdd-8862-65ee2d425690_isrForCrosstalkSources_999140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/5c77d690-6cb9-4fdd-8862-65ee2d425690_isrForCrosstalkSources_999140_53.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash f1ed7613dfa6c938227891c771b9599b parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b77ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/df48b1e4-d0c3-46f0-b79e-bd7015a595d8_isrForCrosstalkSources_999141_42.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/df48b1e4-d0c3-46f0-b79e-bd7015a595d8_isrForCrosstalkSources_999141_42.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash c8226cdc575b78356fb5365d2550e41c parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cbf24905-1b5c-4de3-bd61-d70ef3e61846_isrForCrosstalkSources_999140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cbf24905-1b5c-4de3-bd61-d70ef3e61846_isrForCrosstalkSources_999140_58.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 998b7590b78f8d557cc4eb152d8680a7 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b77d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/7d0b7a61-c089-409c-8ce1-f7e911a2d897_isrForCrosstalkSources_999141_58.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/7d0b7a61-c089-409c-8ce1-f7e911a2d897_isrForCrosstalkSources_999141_58.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash ed9696c276d20d910edb8c8de6cc8f48 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b77ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/2dbb05d8-3c16-4c79-92bb-7b6e69c759c2_isrForCrosstalkSources_999140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/2dbb05d8-3c16-4c79-92bb-7b6e69c759c2_isrForCrosstalkSources_999140_36.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 7c298e53d313d263138920defd10a8fa parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b77c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/42772196-1b05-4251-bf8e-1b4125694504_isrForCrosstalkSources_999141_53.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/42772196-1b05-4251-bf8e-1b4125694504_isrForCrosstalkSources_999141_53.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash ac1cbe4461007f18a132e9bd9730b2a9 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/da27ad95-035e-4511-973c-9cfa8de0cee0_isrForCrosstalkSources_999139_55.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/da27ad95-035e-4511-973c-9cfa8de0cee0_isrForCrosstalkSources_999139_55.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 0e1365bb81f51bd8a4766112eef5498c parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 157 parsl.dataflow.strategy DEBUG: Executor multi has 157 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 39 more blocks parsl.executors.status_handling INFO: Scaling out by 39 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-16 00:03:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-16 00:03:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 00:03:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/71b048a4-91ca-4070-9dbd-a96ec0a04202_isrForCrosstalkSources_999141_44.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/71b048a4-91ca-4070-9dbd-a96ec0a04202_isrForCrosstalkSources_999141_44.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 2103f13326e8c4b80f582a1ed84b4450 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/cb084fea-c526-46ec-93be-565170ef9e13_isrForCrosstalkSources_999141_52.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/cb084fea-c526-46ec-93be-565170ef9e13_isrForCrosstalkSources_999141_52.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 8a6bcd5dfb6531864131ed5ee1a4ad18 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b77e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/cedb1ac5-3a7d-4b0a-80e6-bea2ad1be527_isrForCrosstalkSources_999139_8.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/cedb1ac5-3a7d-4b0a-80e6-bea2ad1be527_isrForCrosstalkSources_999139_8.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash d3bedddb791356068d81b40b8cb17130 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/021d1d95-130c-4e24-8c8d-e5478f5cd68c_isrForCrosstalkSources_999139_26.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/021d1d95-130c-4e24-8c8d-e5478f5cd68c_isrForCrosstalkSources_999139_26.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 7cd2e3793ac74e8c794bcf782020caa2 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f47123f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cb9df948-ae18-42df-a0b8-247ce43d7888_isrForCrosstalkSources_999140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cb9df948-ae18-42df-a0b8-247ce43d7888_isrForCrosstalkSources_999140_9.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash a643f376f0b7881405ba28f47e68d47b parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ef22d0bf-a6bd-4320-88ce-85cd247d387a_isrForCrosstalkSources_999141_46.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ef22d0bf-a6bd-4320-88ce-85cd247d387a_isrForCrosstalkSources_999141_46.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 56db3e9040a4f6e2b1c632fef960bfc5 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45bd9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b5cdcc0e-90a2-4bc7-a659-21034a127550_isrForCrosstalkSources_999140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b5cdcc0e-90a2-4bc7-a659-21034a127550_isrForCrosstalkSources_999140_42.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 03483d768203035c0569940f2f9fd4f5 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45bd98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e093bfa0-1d54-44fe-8f19-047d4724718e_isrForCrosstalkSources_999140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e093bfa0-1d54-44fe-8f19-047d4724718e_isrForCrosstalkSources_999140_34.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 6a4746f358a0ad6137ee70a4108ef0df parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/d339dac6-95e9-4af5-9a37-4df036cd71dc_isrForCrosstalkSources_999140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/d339dac6-95e9-4af5-9a37-4df036cd71dc_isrForCrosstalkSources_999140_43.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 12bce6b9f8d96e10e8329ff78516fff8 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b75e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/63c97003-2994-443d-9856-b051fced3fbb_isrForCrosstalkSources_999140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/63c97003-2994-443d-9856-b051fced3fbb_isrForCrosstalkSources_999140_11.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 9b0248207a69133844092ae09fc74e29 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c4b1e4d1-2abb-4180-bb6b-bd954d0b098b_isrForCrosstalkSources_999140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c4b1e4d1-2abb-4180-bb6b-bd954d0b098b_isrForCrosstalkSources_999140_38.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash ff24e4deeca24b12cdbb163feee9c8f0 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/308d8920-6d8f-4980-8cf5-9a63255de9b7_isrForCrosstalkSources_999141_59.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/308d8920-6d8f-4980-8cf5-9a63255de9b7_isrForCrosstalkSources_999141_59.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash a0fa624649b76e36b06c12f233320258 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/28c61392-0104-4ff5-9e2d-15786f31e012_isrForCrosstalkSources_999140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/28c61392-0104-4ff5-9e2d-15786f31e012_isrForCrosstalkSources_999140_47.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash e204217d5b2d420c9993f1465292dfce parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/de9d7964-c05c-477e-aaab-7929eb1991a4_isrForCrosstalkSources_999141_5.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/de9d7964-c05c-477e-aaab-7929eb1991a4_isrForCrosstalkSources_999141_5.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 92d49d549f1153997679027c5367a60b parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/da1f6e40-ac56-4b33-bdf3-3c5c861c5ffc_isrForCrosstalkSources_999140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/da1f6e40-ac56-4b33-bdf3-3c5c861c5ffc_isrForCrosstalkSources_999140_22.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 371ca17966531e14ed26ba2ec1310a05 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/be0a56da-1882-4750-a268-d17ece32a71c_isrForCrosstalkSources_999141_25.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/be0a56da-1882-4750-a268-d17ece32a71c_isrForCrosstalkSources_999141_25.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 6288e5dd2894faf923a74cca3ee286e8 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1c0027bb-83a6-4d22-b84c-63ff8d5eeeb4_isrForCrosstalkSources_999139_13.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1c0027bb-83a6-4d22-b84c-63ff8d5eeeb4_isrForCrosstalkSources_999139_13.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 5fe3e69238eb6b71d2790017325f7fd4 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e3f9664f-cd0a-494d-8317-685911615730_isrForCrosstalkSources_999141_11.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e3f9664f-cd0a-494d-8317-685911615730_isrForCrosstalkSources_999141_11.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 6dc5705d086d6116ae16837dced74ca9 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b768e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f782a802-c524-4b1d-8b44-b6b770b40cb4_isrForCrosstalkSources_999141_6.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f782a802-c524-4b1d-8b44-b6b770b40cb4_isrForCrosstalkSources_999141_6.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash e3594404e6589cf5e639edabc35b625d parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5010511d-0f0e-49b2-aaf0-51ece2c01c36_isrForCrosstalkSources_999139_60.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5010511d-0f0e-49b2-aaf0-51ece2c01c36_isrForCrosstalkSources_999139_60.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 0190fb4d28bfd868c9159af1a6161f5e parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4bd292cf-bfb4-407e-b140-63fb2c7c03d9_isrForCrosstalkSources_999139_58.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4bd292cf-bfb4-407e-b140-63fb2c7c03d9_isrForCrosstalkSources_999139_58.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 7d302be1d536891872d2169d482a54d5 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/8b82ffb6-7649-4fbb-b1a8-adb6e133e5ed_isrForCrosstalkSources_999140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/8b82ffb6-7649-4fbb-b1a8-adb6e133e5ed_isrForCrosstalkSources_999140_61.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 306aaedaabe3fae9efcc2619dba7f8f4 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/50dfc65c-8dc8-4a10-926f-7d41c24fa557_isrForCrosstalkSources_999141_43.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/50dfc65c-8dc8-4a10-926f-7d41c24fa557_isrForCrosstalkSources_999141_43.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 2e114b5080de81a3fa6bc5b44235173a parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c07ff830-bcb1-447e-a492-446b761aa7cc_isrForCrosstalkSources_999140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c07ff830-bcb1-447e-a492-446b761aa7cc_isrForCrosstalkSources_999140_52.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 7c4b87bb280b485476af4ee473472599 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f0c8b8ca-0d96-4715-b6ea-103c559db310_isrForCrosstalkSources_999141_32.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f0c8b8ca-0d96-4715-b6ea-103c559db310_isrForCrosstalkSources_999141_32.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 9fa33a5aaeda14244a128cf364809d03 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/c4d00cff-512b-409b-9fbb-1fc46ab41aec_isrForCrosstalkSources_999141_17.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/c4d00cff-512b-409b-9fbb-1fc46ab41aec_isrForCrosstalkSources_999141_17.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash af9d3b34bc160dab0c125e3c476f4a2e parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b76fd1a9-1517-419e-a654-b675456978d9_isrForCrosstalkSources_999140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b76fd1a9-1517-419e-a654-b675456978d9_isrForCrosstalkSources_999140_12.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash ef22a5fb1503e638cc206e8102a78829 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/890499ac-419f-4739-86fa-be6d7d62900f_isrForCrosstalkSources_999139_11.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/890499ac-419f-4739-86fa-be6d7d62900f_isrForCrosstalkSources_999139_11.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash adf5f9ffc3ed8cdd1a969ab652b9c593 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149f45b76de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fed34728-00a3-4b15-be69-bb7624050711_isrForCrosstalkSources_999141_30.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fed34728-00a3-4b15-be69-bb7624050711_isrForCrosstalkSources_999141_30.stderr 2024-03-16 00:03:18 proc_lsst.multi:170 [INFO] [multi] job_id 17096695 proc_lsst.multi INFO: [multi] job_id 17096695 2024-03-16 00:03:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096695 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096695 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17096695 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-16 00:03:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-16 00:03:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:18 proc_lsst.multi:170 [INFO] [multi] job_id 17096696 proc_lsst.multi INFO: [multi] job_id 17096696 2024-03-16 00:03:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:03:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096696 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096696 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17096696 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-16 00:03:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-16 00:03:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:03:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:18 proc_lsst.multi:170 [INFO] [multi] job_id 17096697 proc_lsst.multi INFO: [multi] job_id 17096697 2024-03-16 00:03:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:03:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096697 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096697 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17096697 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-16 00:03:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-16 00:03:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:03:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:19 proc_lsst.multi:170 [INFO] [multi] job_id 17096698 proc_lsst.multi INFO: [multi] job_id 17096698 2024-03-16 00:03:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:03:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096698 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096698 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17096698 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-16 00:03:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-16 00:03:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:03:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:19 proc_lsst.multi:170 [INFO] [multi] job_id 17096699 proc_lsst.multi INFO: [multi] job_id 17096699 2024-03-16 00:03:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096699 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096699 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17096699 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-16 00:03:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-16 00:03:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:19 proc_lsst.multi:170 [INFO] [multi] job_id 17096700 proc_lsst.multi INFO: [multi] job_id 17096700 2024-03-16 00:03:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 00:03:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096700 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096700 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17096700 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-16 00:03:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-16 00:03:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 00:03:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:20 proc_lsst.multi:170 [INFO] [multi] job_id 17096701 proc_lsst.multi INFO: [multi] job_id 17096701 2024-03-16 00:03:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 00:03:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096701 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096701 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17096701 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-16 00:03:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-16 00:03:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 00:03:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:20 proc_lsst.multi:170 [INFO] [multi] job_id 17096702 proc_lsst.multi INFO: [multi] job_id 17096702 2024-03-16 00:03:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 00:03:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096702 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096702 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17096702 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-16 00:03:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-16 00:03:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 00:03:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:21 proc_lsst.multi:170 [INFO] [multi] job_id 17096703 proc_lsst.multi INFO: [multi] job_id 17096703 2024-03-16 00:03:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 00:03:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096703 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096703 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17096703 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-16 00:03:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-16 00:03:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 00:03:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:21 proc_lsst.multi:170 [INFO] [multi] job_id 17096704 proc_lsst.multi INFO: [multi] job_id 17096704 2024-03-16 00:03:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-16 00:03:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096704 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096704 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17096704 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-16 00:03:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-16 00:03:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-16 00:03:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:22 proc_lsst.multi:170 [INFO] [multi] job_id 17096705 proc_lsst.multi INFO: [multi] job_id 17096705 2024-03-16 00:03:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-16 00:03:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096705 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096705 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17096705 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-16 00:03:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-16 00:03:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-16 00:03:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:22 proc_lsst.multi:170 [INFO] [multi] job_id 17096706 proc_lsst.multi INFO: [multi] job_id 17096706 2024-03-16 00:03:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-16 00:03:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096706 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096706 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17096706 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-16 00:03:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-16 00:03:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-16 00:03:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:22 proc_lsst.multi:170 [INFO] [multi] job_id 17096707 proc_lsst.multi INFO: [multi] job_id 17096707 2024-03-16 00:03:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-16 00:03:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096707 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096707 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17096707 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-16 00:03:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-16 00:03:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-16 00:03:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:23 proc_lsst.multi:170 [INFO] [multi] job_id 17096708 proc_lsst.multi INFO: [multi] job_id 17096708 2024-03-16 00:03:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-16 00:03:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096708 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096708 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17096708 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-16 00:03:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-16 00:03:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-16 00:03:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:23 proc_lsst.multi:170 [INFO] [multi] job_id 17096709 proc_lsst.multi INFO: [multi] job_id 17096709 2024-03-16 00:03:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-16 00:03:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096709 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096709 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17096709 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-16 00:03:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-16 00:03:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-16 00:03:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:23 proc_lsst.multi:170 [INFO] [multi] job_id 17096710 proc_lsst.multi INFO: [multi] job_id 17096710 2024-03-16 00:03:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-16 00:03:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096710 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096710 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17096710 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-16 00:03:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-16 00:03:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-16 00:03:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:23 proc_lsst.multi:170 [INFO] [multi] job_id 17096711 proc_lsst.multi INFO: [multi] job_id 17096711 2024-03-16 00:03:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-16 00:03:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096711 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096711 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17096711 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-16 00:03:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-16 00:03:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-16 00:03:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:24 proc_lsst.multi:170 [INFO] [multi] job_id 17096712 proc_lsst.multi INFO: [multi] job_id 17096712 2024-03-16 00:03:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-16 00:03:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096712 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096712 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17096712 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-16 00:03:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-16 00:03:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-16 00:03:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:24 proc_lsst.multi:170 [INFO] [multi] job_id 17096713 proc_lsst.multi INFO: [multi] job_id 17096713 2024-03-16 00:03:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-16 00:03:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096713 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096713 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17096713 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-16 00:03:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-16 00:03:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-16 00:03:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:24 proc_lsst.multi:170 [INFO] [multi] job_id 17096714 proc_lsst.multi INFO: [multi] job_id 17096714 2024-03-16 00:03:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-16 00:03:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096714 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096714 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17096714 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-16 00:03:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-16 00:03:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-16 00:03:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:25 proc_lsst.multi:170 [INFO] [multi] job_id 17096715 proc_lsst.multi INFO: [multi] job_id 17096715 2024-03-16 00:03:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-16 00:03:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096715 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096715 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17096715 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-16 00:03:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-16 00:03:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-16 00:03:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:25 proc_lsst.multi:170 [INFO] [multi] job_id 17096716 proc_lsst.multi INFO: [multi] job_id 17096716 2024-03-16 00:03:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-16 00:03:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096716 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096716 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17096716 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-16 00:03:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-16 00:03:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-16 00:03:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:26 proc_lsst.multi:170 [INFO] [multi] job_id 17096717 proc_lsst.multi INFO: [multi] job_id 17096717 2024-03-16 00:03:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-16 00:03:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096717 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096717 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17096717 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-16 00:03:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-16 00:03:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-16 00:03:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:26 proc_lsst.multi:170 [INFO] [multi] job_id 17096718 proc_lsst.multi INFO: [multi] job_id 17096718 2024-03-16 00:03:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-16 00:03:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096718 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096718 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17096718 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-16 00:03:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-16 00:03:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-16 00:03:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:26 proc_lsst.multi:170 [INFO] [multi] job_id 17096719 proc_lsst.multi INFO: [multi] job_id 17096719 2024-03-16 00:03:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-16 00:03:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096719 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096719 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17096719 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-16 00:03:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-16 00:03:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-16 00:03:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:27 proc_lsst.multi:170 [INFO] [multi] job_id 17096720 proc_lsst.multi INFO: [multi] job_id 17096720 2024-03-16 00:03:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-16 00:03:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096720 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096720 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17096720 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-16 00:03:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-16 00:03:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-16 00:03:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:27 proc_lsst.multi:170 [INFO] [multi] job_id 17096721 proc_lsst.multi INFO: [multi] job_id 17096721 2024-03-16 00:03:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-16 00:03:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096721 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096721 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17096721 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-16 00:03:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-16 00:03:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-16 00:03:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:27 proc_lsst.multi:170 [INFO] [multi] job_id 17096722 proc_lsst.multi INFO: [multi] job_id 17096722 2024-03-16 00:03:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-16 00:03:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096722 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096722 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17096722 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-16 00:03:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-16 00:03:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-16 00:03:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a66dc317568217f850e933e7659b7209 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b7762167-4775-4ac3-85a8-06b6dd7887b7_isrForCrosstalkSources_999140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b7762167-4775-4ac3-85a8-06b6dd7887b7_isrForCrosstalkSources_999140_44.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 154c11a7fee8d21872957797b6b7235a with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/201b96f8-1193-4c78-a58a-6cad074af834_isrForCrosstalkSources_999139_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/201b96f8-1193-4c78-a58a-6cad074af834_isrForCrosstalkSources_999139_54.stderr 2024-03-16 00:03:28 proc_lsst.multi:170 [INFO] [multi] job_id 17096723 proc_lsst.multi INFO: [multi] job_id 17096723 2024-03-16 00:03:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-16 00:03:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096723 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096723 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17096723 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-16 00:03:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-16 00:03:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-16 00:03:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:28 proc_lsst.multi:170 [INFO] [multi] job_id 17096724 proc_lsst.multi INFO: [multi] job_id 17096724 2024-03-16 00:03:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096724 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096724 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17096724 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-16 00:03:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-16 00:03:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 00:03:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:29 proc_lsst.multi:170 [INFO] [multi] job_id 17096725 proc_lsst.multi INFO: [multi] job_id 17096725 2024-03-16 00:03:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:29 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096725 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096725 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17096725 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-16 00:03:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-16 00:03:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:29 proc_lsst.multi:170 [INFO] [multi] job_id 17096726 proc_lsst.multi INFO: [multi] job_id 17096726 2024-03-16 00:03:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:03:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:29 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096726 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096726 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17096726 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-16 00:03:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-16 00:03:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:03:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:30 proc_lsst.multi:170 [INFO] [multi] job_id 17096727 proc_lsst.multi INFO: [multi] job_id 17096727 2024-03-16 00:03:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:03:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:30 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096727 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096727 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17096727 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-16 00:03:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-16 00:03:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:03:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:30 proc_lsst.multi:170 [INFO] [multi] job_id 17096728 proc_lsst.multi INFO: [multi] job_id 17096728 2024-03-16 00:03:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:03:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:30 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096728 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096728 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17096728 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-16 00:03:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-16 00:03:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:03:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:31 proc_lsst.multi:170 [INFO] [multi] job_id 17096729 proc_lsst.multi INFO: [multi] job_id 17096729 2024-03-16 00:03:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:31 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096729 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096729 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17096729 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-16 00:03:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-16 00:03:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 00:03:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:31 proc_lsst.multi:170 [INFO] [multi] job_id 17096730 proc_lsst.multi INFO: [multi] job_id 17096730 2024-03-16 00:03:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:31 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096730 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096730 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17096730 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-16 00:03:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-16 00:03:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:32 proc_lsst.multi:170 [INFO] [multi] job_id 17096731 proc_lsst.multi INFO: [multi] job_id 17096731 2024-03-16 00:03:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:03:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:32 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096731 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096731 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17096731 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-16 00:03:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:32 proc_lsst.multi:170 [INFO] [multi] job_id 17096732 proc_lsst.multi INFO: [multi] job_id 17096732 2024-03-16 00:03:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:03:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:32 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096732 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096732 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17096732 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-16 00:03:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:32 proc_lsst.multi:170 [INFO] [multi] job_id 17096733 proc_lsst.multi INFO: [multi] job_id 17096733 2024-03-16 00:03:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:03:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:32 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096733 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096733 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17096733 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22674326706576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 40, active_tasks = 184 parsl.dataflow.strategy DEBUG: Executor multi has 184 active tasks, 1/39 running/pending blocks, and 12 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 40 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 6 more blocks parsl.executors.status_handling INFO: Scaling out by 6 blocks parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-16 00:03:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:03:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:33 proc_lsst.multi:170 [INFO] [multi] job_id 17096734 proc_lsst.multi INFO: [multi] job_id 17096734 2024-03-16 00:03:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:33 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096734 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096734 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17096734 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-16 00:03:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:33 proc_lsst.multi:170 [INFO] [multi] job_id 17096735 proc_lsst.multi INFO: [multi] job_id 17096735 2024-03-16 00:03:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 00:03:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:33 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096735 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096735 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17096735 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-16 00:03:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:33 proc_lsst.multi:170 [INFO] [multi] job_id 17096736 proc_lsst.multi INFO: [multi] job_id 17096736 2024-03-16 00:03:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 00:03:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:33 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096736 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096736 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17096736 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-16 00:03:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 00:03:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:34 proc_lsst.multi:170 [INFO] [multi] job_id 17096737 proc_lsst.multi INFO: [multi] job_id 17096737 2024-03-16 00:03:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 00:03:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096737 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096737 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17096737 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-16 00:03:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-16 00:03:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 00:03:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:34 proc_lsst.multi:170 [INFO] [multi] job_id 17096738 proc_lsst.multi INFO: [multi] job_id 17096738 2024-03-16 00:03:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 00:03:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096738 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096738 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17096738 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-16 00:03:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-16 00:03:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:03:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:03:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:03:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:03:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:03:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:03:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:03:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:03:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 00:03:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,n3009,127.0.0.1,169.254.95.120,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54817 --result_port=54792 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:03:34 proc_lsst.multi:170 [INFO] [multi] job_id 17096739 proc_lsst.multi INFO: [multi] job_id 17096739 2024-03-16 00:03:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-16 00:03:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:03:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096739 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096739 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17096739 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22674326706576 parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 052623726f29d11276706feae4d5f94a with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/eb8625f4-240f-4ea7-9415-93eb5dc0be8f_isrForCrosstalkSources_999140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/eb8625f4-240f-4ea7-9415-93eb5dc0be8f_isrForCrosstalkSources_999140_31.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 183 parsl.dataflow.strategy DEBUG: Executor multi has 183 active tasks, 1/45 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22674326706576 parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19fa902d223af78b70008a601fe050ac with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a350945b-db69-4882-84b9-ec8ba298a1c9_isrForCrosstalkSources_999141_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a350945b-db69-4882-84b9-ec8ba298a1c9_isrForCrosstalkSources_999141_49.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 182 parsl.dataflow.strategy DEBUG: Executor multi has 182 active tasks, 1/45 running/pending blocks, and 47 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22674326706576 parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9708fb2b930864f0ebb481f64ff1be5e with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a4f15022-2020-40ad-918d-88a7d87b48c8_isrForCrosstalkSources_999139_31.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a4f15022-2020-40ad-918d-88a7d87b48c8_isrForCrosstalkSources_999139_31.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a44a7942e48bb11e590540b3804d6330 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c10aa9b2-3aa0-44af-9e65-56766b06c4b3_isrForCrosstalkSources_999140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c10aa9b2-3aa0-44af-9e65-56766b06c4b3_isrForCrosstalkSources_999140_48.stderr 2024-03-16 00:03:47 proc_lsst.multi:146 [INFO] found job 13819 in provider local proc_lsst.multi INFO: found job 13819 in provider local 2024-03-16 00:03:48 proc_lsst.multi:146 [INFO] found job 17096695 in provider astro proc_lsst.multi INFO: found job 17096695 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:49 proc_lsst.multi:146 [INFO] found job 17096696 in provider astro proc_lsst.multi INFO: found job 17096696 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:50 proc_lsst.multi:146 [INFO] found job 17096697 in provider astro proc_lsst.multi INFO: found job 17096697 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:50 proc_lsst.multi:146 [INFO] found job 17096698 in provider astro proc_lsst.multi INFO: found job 17096698 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:51 proc_lsst.multi:146 [INFO] found job 17096699 in provider astro proc_lsst.multi INFO: found job 17096699 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:52 proc_lsst.multi:146 [INFO] found job 17096700 in provider astro proc_lsst.multi INFO: found job 17096700 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:53 proc_lsst.multi:146 [INFO] found job 17096701 in provider astro proc_lsst.multi INFO: found job 17096701 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8cbcded28aea598261742fbc868d736 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/f167852d-1030-47c0-b471-b16bbeecdc37_isrForCrosstalkSources_999140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/f167852d-1030-47c0-b471-b16bbeecdc37_isrForCrosstalkSources_999140_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:53 proc_lsst.multi:146 [INFO] found job 17096702 in provider astro proc_lsst.multi INFO: found job 17096702 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ef64a81d3ed7f5664909fb0304579d0 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/045c4ef9-44ec-4610-ba4a-7c01d1a721f2_isrForCrosstalkSources_999140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/045c4ef9-44ec-4610-ba4a-7c01d1a721f2_isrForCrosstalkSources_999140_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:54 proc_lsst.multi:146 [INFO] found job 17096703 in provider astro proc_lsst.multi INFO: found job 17096703 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:55 proc_lsst.multi:146 [INFO] found job 17096704 in provider astro proc_lsst.multi INFO: found job 17096704 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:56 proc_lsst.multi:146 [INFO] found job 17096705 in provider astro proc_lsst.multi INFO: found job 17096705 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:56 proc_lsst.multi:146 [INFO] found job 17096706 in provider astro proc_lsst.multi INFO: found job 17096706 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:57 proc_lsst.multi:146 [INFO] found job 17096707 in provider astro proc_lsst.multi INFO: found job 17096707 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:03:58 proc_lsst.multi:146 [INFO] found job 17096708 in provider astro proc_lsst.multi INFO: found job 17096708 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:00 proc_lsst.multi:146 [INFO] found job 17096709 in provider astro proc_lsst.multi INFO: found job 17096709 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:00 proc_lsst.multi:146 [INFO] found job 17096710 in provider astro proc_lsst.multi INFO: found job 17096710 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:01 proc_lsst.multi:146 [INFO] found job 17096711 in provider astro proc_lsst.multi INFO: found job 17096711 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b1cbd828344006390d0932067a6ef66 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ee8e4df8-a099-4375-9b6f-fb6ed0559c90_isrForCrosstalkSources_999141_47.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ee8e4df8-a099-4375-9b6f-fb6ed0559c90_isrForCrosstalkSources_999141_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:02 proc_lsst.multi:146 [INFO] found job 17096712 in provider astro proc_lsst.multi INFO: found job 17096712 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dd43c4dbaebe25881210bc274e84a63 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/77dd27a8-d7d7-403b-b6a4-25509ce9c21a_isrForCrosstalkSources_999140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/77dd27a8-d7d7-403b-b6a4-25509ce9c21a_isrForCrosstalkSources_999140_8.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9316651e21cc5f0a225e2ff126a3c9e9 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a035f6bd-91e3-475c-ac90-a693fe002efc_isrForCrosstalkSources_999141_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a035f6bd-91e3-475c-ac90-a693fe002efc_isrForCrosstalkSources_999141_26.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 679c261fab2ecf033bda101bf17d73b3 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/480f22d6-639f-472a-a3fc-47b9c9c69304_isrForCrosstalkSources_999139_14.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/480f22d6-639f-472a-a3fc-47b9c9c69304_isrForCrosstalkSources_999139_14.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7434ba3afa612dffc8927cc642ce4d66 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5e026b0d-e61b-41ce-9c7c-201a79ae73e0_isrForCrosstalkSources_999139_20.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5e026b0d-e61b-41ce-9c7c-201a79ae73e0_isrForCrosstalkSources_999139_20.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0913067b49a39ab4e6674e0130b3c1de with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/28dac436-8ec7-454d-8f7a-7f1f236ad1fd_isrForCrosstalkSources_999140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/28dac436-8ec7-454d-8f7a-7f1f236ad1fd_isrForCrosstalkSources_999140_40.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52bf9a7f3b0519c2ae3b05560baea28f with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e9ecf5c5-3bf5-4fce-8583-fd8e9790e3c4_isrForCrosstalkSources_999141_24.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e9ecf5c5-3bf5-4fce-8583-fd8e9790e3c4_isrForCrosstalkSources_999141_24.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 023a7d927ddf9518b7b4a3780ca9260d with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/90d99ddf-79b0-4537-9cd2-5a5a500e4e8c_isrForCrosstalkSources_999141_20.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/90d99ddf-79b0-4537-9cd2-5a5a500e4e8c_isrForCrosstalkSources_999141_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:02 proc_lsst.multi:146 [INFO] found job 17096713 in provider astro proc_lsst.multi INFO: found job 17096713 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb1bf2836333978209bd33f9f944655 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9d2776ef-62eb-410f-ae9e-e2f60e230dc0_isrForCrosstalkSources_999140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9d2776ef-62eb-410f-ae9e-e2f60e230dc0_isrForCrosstalkSources_999140_21.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3784491c05ea3e5b799bd2398443099e with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/32cbe50e-f9e3-41f8-a88f-4d047bfe5a95_isrForCrosstalkSources_999140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/32cbe50e-f9e3-41f8-a88f-4d047bfe5a95_isrForCrosstalkSources_999140_16.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93b4048d20528c9782a7dd43fc0c49ef with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/f3939eb8-9910-40be-a91f-3547c37c1c30_isrForCrosstalkSources_999139_15.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/f3939eb8-9910-40be-a91f-3547c37c1c30_isrForCrosstalkSources_999139_15.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 114baf1a3f61021599edf3f8de1b8390 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/90a45290-a0d8-4f28-a08c-df587f1d7aa3_isrForCrosstalkSources_999141_10.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/90a45290-a0d8-4f28-a08c-df587f1d7aa3_isrForCrosstalkSources_999141_10.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9712403899ecafd869e9e7efeb6c829 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/0c5e44ec-889e-49a7-ae80-a7f15192f4cb_isrForCrosstalkSources_999141_16.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/0c5e44ec-889e-49a7-ae80-a7f15192f4cb_isrForCrosstalkSources_999141_16.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a907880b0461b59543ba572a515091c with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1c4a5719-4c27-4e95-8fa1-548a15b1e55d_isrForCrosstalkSources_999139_30.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1c4a5719-4c27-4e95-8fa1-548a15b1e55d_isrForCrosstalkSources_999139_30.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 782721ab127e5b37a330867fe1b972ae with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4e0717d8-a768-4979-a991-e68b3b28422e_isrForCrosstalkSources_999139_28.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4e0717d8-a768-4979-a991-e68b3b28422e_isrForCrosstalkSources_999139_28.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8ea1ac2e28ba4fe508747879f47ebfb with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b0d4850f-2f24-49d1-9edc-729e93b38d5b_isrForCrosstalkSources_999140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b0d4850f-2f24-49d1-9edc-729e93b38d5b_isrForCrosstalkSources_999140_6.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5073575a66bab2100ecf154daf345815 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/245e27df-37cc-4816-b7d8-c9174b3c2fb2_isrForCrosstalkSources_999139_45.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/245e27df-37cc-4816-b7d8-c9174b3c2fb2_isrForCrosstalkSources_999139_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:03 proc_lsst.multi:146 [INFO] found job 17096714 in provider astro proc_lsst.multi INFO: found job 17096714 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89a0739db896462509efc3e627a75e01 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/be36e612-b292-4314-a8b4-a81e2999a78f_isrForCrosstalkSources_999141_36.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/be36e612-b292-4314-a8b4-a81e2999a78f_isrForCrosstalkSources_999141_36.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c37b094ad8701abb820f3e9bba63acb with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/54e62f14-4873-4a0e-80e7-ceeba6a615e0_isrForCrosstalkSources_999140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/54e62f14-4873-4a0e-80e7-ceeba6a615e0_isrForCrosstalkSources_999140_41.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19cd730ea8161a08a3bdf33bbb9c28d7 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4fd12186-1f06-496c-b2dd-ba85b31b78fb_isrForCrosstalkSources_999140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4fd12186-1f06-496c-b2dd-ba85b31b78fb_isrForCrosstalkSources_999140_30.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d1d696c2d268efe04de3d435fd38d3 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/9441188f-d74c-4cf1-ba06-84cdbf874857_isrForCrosstalkSources_999141_28.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/9441188f-d74c-4cf1-ba06-84cdbf874857_isrForCrosstalkSources_999141_28.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5d89575edb6db8bb4953c9cd3132d78 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/7f222673-a548-417e-8d60-f3b134b497ff_isrForCrosstalkSources_999141_31.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/7f222673-a548-417e-8d60-f3b134b497ff_isrForCrosstalkSources_999141_31.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f32b85d1f8325d2fb9b570c8d5c18f3 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/ba8b528d-4e69-46fd-a155-392181459c50_isrForCrosstalkSources_999139_9.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/ba8b528d-4e69-46fd-a155-392181459c50_isrForCrosstalkSources_999139_9.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 128c78f1e8dade86390484b63a8d5786 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7456046a-0423-4d8e-881a-7d30f15d097b_isrForCrosstalkSources_999139_52.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7456046a-0423-4d8e-881a-7d30f15d097b_isrForCrosstalkSources_999139_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:03 proc_lsst.multi:146 [INFO] found job 17096715 in provider astro proc_lsst.multi INFO: found job 17096715 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef4e253197c5652b073c0315412b92bb with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/d4b48fe2-c9d0-4096-a267-615b3c9b498f_isrForCrosstalkSources_999140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/d4b48fe2-c9d0-4096-a267-615b3c9b498f_isrForCrosstalkSources_999140_28.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 012e7dd1204dd70c69d69c0b9fd274f0 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/150c83a2-bd37-4e85-bfa4-17e4df157c1b_isrForCrosstalkSources_999139_19.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/150c83a2-bd37-4e85-bfa4-17e4df157c1b_isrForCrosstalkSources_999139_19.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b5e8db95a329f721d18243cae3db9ed with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/950bfe16-8f0c-49c1-a803-dc7f462daa29_isrForCrosstalkSources_999141_8.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/950bfe16-8f0c-49c1-a803-dc7f462daa29_isrForCrosstalkSources_999141_8.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81562b5eb8484820d09d6feead50df48 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/460c7d69-6adb-484b-af05-a51323fec3a7_isrForCrosstalkSources_999141_45.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/460c7d69-6adb-484b-af05-a51323fec3a7_isrForCrosstalkSources_999141_45.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 639dd9f5504a4fb93d7cb025b9b65efa with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/ee470195-949b-4c54-a239-12fb17af9d46_isrForCrosstalkSources_999140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/ee470195-949b-4c54-a239-12fb17af9d46_isrForCrosstalkSources_999140_33.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5da0aa29cfdb7b7422fa03443a86eb26 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/11d8deba-80d9-41d3-ad2f-9b002fefff94_isrForCrosstalkSources_999140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/11d8deba-80d9-41d3-ad2f-9b002fefff94_isrForCrosstalkSources_999140_13.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c2013d9011ab8747791d5ecfba25cad with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a2681153-7484-4b9c-805c-cef18a608d2c_isrForCrosstalkSources_999140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a2681153-7484-4b9c-805c-cef18a608d2c_isrForCrosstalkSources_999140_20.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df072b6d59740f48866c521121a3e643 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cd555ae9-7142-4311-9d6c-57b71641955b_isrForCrosstalkSources_999140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cd555ae9-7142-4311-9d6c-57b71641955b_isrForCrosstalkSources_999140_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:04 proc_lsst.multi:146 [INFO] found job 17096716 in provider astro proc_lsst.multi INFO: found job 17096716 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d427b2c414162afcddb0ec336f33cf1c with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/fa765250-4956-480f-aecb-5e3a9f88ad8b_isrForCrosstalkSources_999139_34.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/fa765250-4956-480f-aecb-5e3a9f88ad8b_isrForCrosstalkSources_999139_34.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87da6b57e0640e7b6430d6967a213612 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/f939bb27-0bd4-4f57-b9cf-38fdf6718d6e_isrForCrosstalkSources_999139_49.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/f939bb27-0bd4-4f57-b9cf-38fdf6718d6e_isrForCrosstalkSources_999139_49.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2df474d3629284a7ae49f53284f5690d with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/721285ea-61ad-4e7b-bf6a-98510cbcce53_isrForCrosstalkSources_999139_4.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/721285ea-61ad-4e7b-bf6a-98510cbcce53_isrForCrosstalkSources_999139_4.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c304f9ed5e88e4dc992f6aca26b217 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2ac714de-3875-4d2f-a5e9-2fa45721448f_isrForCrosstalkSources_999141_39.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2ac714de-3875-4d2f-a5e9-2fa45721448f_isrForCrosstalkSources_999141_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:05 proc_lsst.multi:146 [INFO] found job 17096717 in provider astro proc_lsst.multi INFO: found job 17096717 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea8d29d59944d3a863a68e0477cb3079 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a776cc66-a856-479b-93cc-9f51378fa528_isrForCrosstalkSources_999140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a776cc66-a856-479b-93cc-9f51378fa528_isrForCrosstalkSources_999140_4.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbe95063ee0dce3277f54dcd7f5f06bc with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/905633c6-4b15-4341-9a25-2f35d34c5204_isrForCrosstalkSources_999140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/905633c6-4b15-4341-9a25-2f35d34c5204_isrForCrosstalkSources_999140_17.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20369bfff92e50273a1c3496bc1b1e17 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/75dd38c7-ffdd-4515-9770-63a6fda67d63_isrForCrosstalkSources_999140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/75dd38c7-ffdd-4515-9770-63a6fda67d63_isrForCrosstalkSources_999140_23.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ec325b48d240a1181d6334125404f52 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/adf1ce92-5d02-4333-a679-d906bf85d8a7_isrForCrosstalkSources_999139_62.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/adf1ce92-5d02-4333-a679-d906bf85d8a7_isrForCrosstalkSources_999139_62.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85a3972e566a2b68b92a527782766314 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/745f2920-5d70-4e60-a815-a095c9a20138_isrForCrosstalkSources_999139_50.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/745f2920-5d70-4e60-a815-a095c9a20138_isrForCrosstalkSources_999139_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:06 proc_lsst.multi:146 [INFO] found job 17096718 in provider astro proc_lsst.multi INFO: found job 17096718 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a06d2859ad062e25357dfb4c03ad62d6 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/db532951-f2d3-4650-8118-3e8b8cd8ff63_isrForCrosstalkSources_999139_12.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/db532951-f2d3-4650-8118-3e8b8cd8ff63_isrForCrosstalkSources_999139_12.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c338e122551f314db37a984b60b7a467 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/d906cdcc-cb3f-4860-b6ee-08a52ab654e6_isrForCrosstalkSources_999139_2.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/d906cdcc-cb3f-4860-b6ee-08a52ab654e6_isrForCrosstalkSources_999139_2.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 628d045558474fb43df926210f37fc77 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a503c3a8-0156-4b3c-bc53-48994311200a_isrForCrosstalkSources_999139_7.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a503c3a8-0156-4b3c-bc53-48994311200a_isrForCrosstalkSources_999139_7.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa1c37e87d43fd4aa0b20c0540c2e15 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4db68bae-ab93-432c-9f6c-e01ba03d26c3_isrForCrosstalkSources_999140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4db68bae-ab93-432c-9f6c-e01ba03d26c3_isrForCrosstalkSources_999140_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:06 proc_lsst.multi:146 [INFO] found job 17096719 in provider astro proc_lsst.multi INFO: found job 17096719 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef0eaf2816917579b0d2c9feee6e872b with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/6d3315fb-4017-41a3-b8b2-ffbf4e449ce5_isrForCrosstalkSources_999139_47.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/6d3315fb-4017-41a3-b8b2-ffbf4e449ce5_isrForCrosstalkSources_999139_47.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a35733f86f0780d9338f8deaa1a1a6e7 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/75f5d7fc-9e1e-4bf4-a5ce-0aefd16fac76_isrForCrosstalkSources_999141_60.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/75f5d7fc-9e1e-4bf4-a5ce-0aefd16fac76_isrForCrosstalkSources_999141_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:07 proc_lsst.multi:146 [INFO] found job 17096720 in provider astro proc_lsst.multi INFO: found job 17096720 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:08 proc_lsst.multi:146 [INFO] found job 17096721 in provider astro proc_lsst.multi INFO: found job 17096721 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:10 proc_lsst.multi:146 [INFO] found job 17096722 in provider astro proc_lsst.multi INFO: found job 17096722 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1482628fb4d9269148a9984f5f851b09 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/0d192347-dc69-4c5a-b45e-9ff6d498d1a2_isrForCrosstalkSources_999139_25.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/0d192347-dc69-4c5a-b45e-9ff6d498d1a2_isrForCrosstalkSources_999139_25.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a2c7a6a621cdb0645878e21b33d1dc0 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/d9db02b1-6d3f-42fb-a779-47512785f27b_isrForCrosstalkSources_999139_24.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/d9db02b1-6d3f-42fb-a779-47512785f27b_isrForCrosstalkSources_999139_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:10 proc_lsst.multi:146 [INFO] found job 17096723 in provider astro proc_lsst.multi INFO: found job 17096723 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 763cb84ae48968eabdf969930ac69e09 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/20f73278-e429-4168-b1a9-ef9f4e24b541_isrForCrosstalkSources_999140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/20f73278-e429-4168-b1a9-ef9f4e24b541_isrForCrosstalkSources_999140_55.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ef39f11f6e7a12ffe46c169d2ce421 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/c33fd977-ceb9-4df3-ad0b-011c3ce6acbb_isrForCrosstalkSources_999139_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/c33fd977-ceb9-4df3-ad0b-011c3ce6acbb_isrForCrosstalkSources_999139_18.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0caf446dd7474e1c3dfc51d182188781 with result from task 62 parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:11 proc_lsst.multi:146 [INFO] found job 17096724 in provider astro proc_lsst.multi INFO: found job 17096724 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/384b6f44-b6fa-4b2b-98c5-535918ee4d2f_isrForCrosstalkSources_999140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/384b6f44-b6fa-4b2b-98c5-535918ee4d2f_isrForCrosstalkSources_999140_60.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8deba3271f82ee487a7760cf894de3f with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/1c298443-a8c5-4579-8ef1-6c482b5ff0de_isrForCrosstalkSources_999141_2.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/1c298443-a8c5-4579-8ef1-6c482b5ff0de_isrForCrosstalkSources_999141_2.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e0198462da567f9916f05c82d316d4e with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/c0419f8e-1873-4513-ae81-a71a55d839ec_isrForCrosstalkSources_999141_27.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/c0419f8e-1873-4513-ae81-a71a55d839ec_isrForCrosstalkSources_999141_27.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1c2f1f2941aed8569de398a26a77c3c with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/14826200-eb2f-40ac-a53a-c46dcc8c9d27_isrForCrosstalkSources_999141_15.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/14826200-eb2f-40ac-a53a-c46dcc8c9d27_isrForCrosstalkSources_999141_15.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7b3672fedcb731a7cabef3acb255270 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/faf637b9-7e9a-4d02-b652-5f3b152b026d_isrForCrosstalkSources_999140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/faf637b9-7e9a-4d02-b652-5f3b152b026d_isrForCrosstalkSources_999140_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:12 proc_lsst.multi:146 [INFO] found job 17096725 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096725 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cb92b3fd2aea6a11b37f7e5a7033086 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/468eda4d-f8ec-439f-a43f-15dc834c4dba_isrForCrosstalkSources_999141_1.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/468eda4d-f8ec-439f-a43f-15dc834c4dba_isrForCrosstalkSources_999141_1.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86244beff8ed745c561c9c27daecc605 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/0284ffe9-97f6-4780-8614-83234540c87f_isrForCrosstalkSources_999141_38.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/0284ffe9-97f6-4780-8614-83234540c87f_isrForCrosstalkSources_999141_38.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55a3862630ea922d5f3e9aa1e6b71d5b with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/daf44ad0-02ff-45be-b32f-0cfde0aa424f_isrForCrosstalkSources_999139_61.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/daf44ad0-02ff-45be-b32f-0cfde0aa424f_isrForCrosstalkSources_999139_61.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a231ac167d7c14edc79ce348316963b4 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/111470a8-9adb-47c9-8d50-a0d7bf99ce56_isrForCrosstalkSources_999139_43.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/111470a8-9adb-47c9-8d50-a0d7bf99ce56_isrForCrosstalkSources_999139_43.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92aa8252c70e2ea48781021aca5d616b with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/099a7248-1d3b-46c2-965a-ca2e6d6c9c9a_isrForCrosstalkSources_999141_9.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/099a7248-1d3b-46c2-965a-ca2e6d6c9c9a_isrForCrosstalkSources_999141_9.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 799a9b719fae0bba497691e225963a46 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/6936e32a-d665-41b3-9b35-1f22338cfaf0_isrForCrosstalkSources_999141_12.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/6936e32a-d665-41b3-9b35-1f22338cfaf0_isrForCrosstalkSources_999141_12.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5103ed2e71163d5cbe772e38904ea732 with result from task 70 parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:12 proc_lsst.multi:146 [INFO] found job 17096726 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096726 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cce00877-e37b-4d94-baf6-b1ae9deac7da_isrForCrosstalkSources_999140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cce00877-e37b-4d94-baf6-b1ae9deac7da_isrForCrosstalkSources_999140_14.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0762d2219bf87677f3c83bb3a3a2a480 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c0480bc6-a5ed-40a5-9081-410247382973_isrForCrosstalkSources_999140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c0480bc6-a5ed-40a5-9081-410247382973_isrForCrosstalkSources_999140_50.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6f3ed5829569eda8901ea2ef2d30301 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a3c2c2f5-67ea-48c2-8520-b55ec3a8e875_isrForCrosstalkSources_999140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a3c2c2f5-67ea-48c2-8520-b55ec3a8e875_isrForCrosstalkSources_999140_56.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0171fb8c3b9d559c093eeb03d3130813 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/75f5c6d5-83b7-452c-a479-fbd129044b45_isrForCrosstalkSources_999140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/75f5c6d5-83b7-452c-a479-fbd129044b45_isrForCrosstalkSources_999140_46.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a59aef93645326808b0f5cea8ec5fb8 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/8adaa531-b601-4194-b0af-2630b26fce08_isrForCrosstalkSources_999139_22.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/8adaa531-b601-4194-b0af-2630b26fce08_isrForCrosstalkSources_999139_22.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11a31073eaf88a073b69eca267ced1a4 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/caa78fa7-cef6-4981-a70e-d8c934c55a9d_isrForCrosstalkSources_999140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/caa78fa7-cef6-4981-a70e-d8c934c55a9d_isrForCrosstalkSources_999140_62.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cef2afe7ba66dae9ba0c0773523239f2 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5107bb27-5471-4e69-bb0d-c1124631a95d_isrForCrosstalkSources_999139_42.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5107bb27-5471-4e69-bb0d-c1124631a95d_isrForCrosstalkSources_999139_42.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d0eac13f54d5eaf7de40ef875fe4792 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/19796626-63fb-4d7e-a26b-58ca5791e2c4_isrForCrosstalkSources_999140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/19796626-63fb-4d7e-a26b-58ca5791e2c4_isrForCrosstalkSources_999140_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:13 proc_lsst.multi:146 [INFO] found job 17096727 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096727 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22cf7f7e705031bb06e98cd82f6b24ee with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9d91fe40-fc9b-4d7f-9c2f-c9e27d5e38a3_isrForCrosstalkSources_999140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9d91fe40-fc9b-4d7f-9c2f-c9e27d5e38a3_isrForCrosstalkSources_999140_54.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80aff07c5f557b590deb1a2cebfbf8ea with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/3a894255-6a1e-4e3a-9bcc-85eb73d20104_isrForCrosstalkSources_999140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/3a894255-6a1e-4e3a-9bcc-85eb73d20104_isrForCrosstalkSources_999140_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:14 proc_lsst.multi:146 [INFO] found job 17096728 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096728 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:14 proc_lsst.multi:146 [INFO] found job 17096729 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096729 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dc805ddc65ba0e60024c2b730b8829e with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/562a6acf-f269-45cf-9d7b-f674f3b2d458_isrForCrosstalkSources_999139_53.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/562a6acf-f269-45cf-9d7b-f674f3b2d458_isrForCrosstalkSources_999139_53.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9f1348748a3ac6eeb1b490daaf33943 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/8a3275f3-0062-47b5-846a-f46d5c92e8d2_isrForCrosstalkSources_999140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/8a3275f3-0062-47b5-846a-f46d5c92e8d2_isrForCrosstalkSources_999140_45.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92ef788e50cf2811ab8589a0cb0af029 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/8a2f3425-8fb6-437b-9cce-58f5cb950aa5_isrForCrosstalkSources_999139_40.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/8a2f3425-8fb6-437b-9cce-58f5cb950aa5_isrForCrosstalkSources_999139_40.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfbc530a4996181e10e03d3f411d25c4 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a179b5ca-7897-4708-8086-b50ddc84466e_isrForCrosstalkSources_999139_23.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a179b5ca-7897-4708-8086-b50ddc84466e_isrForCrosstalkSources_999139_23.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2011039bafbaa3c4bc3e9d9674f38762 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7b15f7c6-877f-4b3d-a841-84528bc65cc8_isrForCrosstalkSources_999139_36.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7b15f7c6-877f-4b3d-a841-84528bc65cc8_isrForCrosstalkSources_999139_36.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1665f6af42f2606383bf352ecc53ef0c with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1a2c6e5f-fd59-4b4e-b589-ba6ae73a4c5a_isrForCrosstalkSources_999139_29.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1a2c6e5f-fd59-4b4e-b589-ba6ae73a4c5a_isrForCrosstalkSources_999139_29.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f76ad3c18714ec8894b657324521bc47 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/313e6afa-dd1a-4fde-8b0b-27a69d4f6bd6_isrForCrosstalkSources_999139_35.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/313e6afa-dd1a-4fde-8b0b-27a69d4f6bd6_isrForCrosstalkSources_999139_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:15 proc_lsst.multi:146 [INFO] found job 17096730 in provider escience_a40 proc_lsst.multi INFO: found job 17096730 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7997d22a5ba0cb94f3c58fb69a1dce2d with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2b6986f2-28ce-4441-bdf9-5a6739049ea1_isrForCrosstalkSources_999141_56.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2b6986f2-28ce-4441-bdf9-5a6739049ea1_isrForCrosstalkSources_999141_56.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ee446058085d66b513f7662683d9d97 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1e8ac794-9eaf-4d45-80ab-218256491469_isrForCrosstalkSources_999139_37.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1e8ac794-9eaf-4d45-80ab-218256491469_isrForCrosstalkSources_999139_37.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df3d3725bc6393844604fee67cf3829d with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f9c2637a-08d5-48f5-824f-100e55eec0e9_isrForCrosstalkSources_999141_22.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f9c2637a-08d5-48f5-824f-100e55eec0e9_isrForCrosstalkSources_999141_22.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79c9af32f0b1c37166c9f9640390bfcb with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fdc31b65-30ae-468a-9520-9b35e1bece48_isrForCrosstalkSources_999141_37.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fdc31b65-30ae-468a-9520-9b35e1bece48_isrForCrosstalkSources_999141_37.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dba80cec8af6ec1bda4f61efba3105f with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/aa7248bd-03fa-478f-8c25-ca10375ac813_isrForCrosstalkSources_999141_57.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/aa7248bd-03fa-478f-8c25-ca10375ac813_isrForCrosstalkSources_999141_57.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96d885ae5ddd25130604512bfc7e4314 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/23868abd-7724-426b-8901-22f21662897f_isrForCrosstalkSources_999139_1.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/23868abd-7724-426b-8901-22f21662897f_isrForCrosstalkSources_999139_1.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e9e2cc68d6e0a05020c2d674c9bc184 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e5cb1055-26c1-41c8-9b7b-c912d143045e_isrForCrosstalkSources_999141_50.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e5cb1055-26c1-41c8-9b7b-c912d143045e_isrForCrosstalkSources_999141_50.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5d94d3806e93e0785fa4580b1ecb9c0 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a70615e1-cb36-40fa-b0b0-3a648e3bf6e8_isrForCrosstalkSources_999139_33.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/a70615e1-cb36-40fa-b0b0-3a648e3bf6e8_isrForCrosstalkSources_999139_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:16 proc_lsst.multi:146 [INFO] found job 17096731 in provider escience_a40 proc_lsst.multi INFO: found job 17096731 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12ac979947d60eaa7e805d3ec3f655e7 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/d4678a23-0282-44d7-8274-946b5465fdf7_isrForCrosstalkSources_999141_18.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/d4678a23-0282-44d7-8274-946b5465fdf7_isrForCrosstalkSources_999141_18.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db8e43975f50913e4d4e5a566b34639 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/50f74c17-75f0-4c90-b327-b9435713d65a_isrForCrosstalkSources_999139_21.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/50f74c17-75f0-4c90-b327-b9435713d65a_isrForCrosstalkSources_999139_21.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c01f1a34695980d89dbe54120c5f9cbc with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/824e063e-5d01-43aa-9963-907f13244389_isrForCrosstalkSources_999141_48.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/824e063e-5d01-43aa-9963-907f13244389_isrForCrosstalkSources_999141_48.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df11abd297ea0f3aceab54e6e328792a with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/01d301ed-813c-40a0-ada3-975317ac83eb_isrForCrosstalkSources_999139_57.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/01d301ed-813c-40a0-ada3-975317ac83eb_isrForCrosstalkSources_999139_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:17 proc_lsst.multi:146 [INFO] found job 17096732 in provider escience_a40 proc_lsst.multi INFO: found job 17096732 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 426f6d8cad8d5ddb90ef305271acc0ae with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cc4d1950-dabe-425b-88ad-a95a35dde9f5_isrForCrosstalkSources_999140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cc4d1950-dabe-425b-88ad-a95a35dde9f5_isrForCrosstalkSources_999140_27.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32c88a14ce362981847e0bc48bfe0a34 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/bcbe899b-1b01-4b01-bd25-f501540a7ac2_isrForCrosstalkSources_999140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/bcbe899b-1b01-4b01-bd25-f501540a7ac2_isrForCrosstalkSources_999140_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:17 proc_lsst.multi:146 [INFO] found job 17096733 in provider escience_a40 proc_lsst.multi INFO: found job 17096733 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66874716acdf67adf6f581d464d51869 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/460a9e9d-845a-400d-840f-da1a00c50888_isrForCrosstalkSources_999141_35.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/460a9e9d-845a-400d-840f-da1a00c50888_isrForCrosstalkSources_999141_35.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b61e0f6efd4fe536ed4e39563c39e59 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cd1413d7-efcd-47da-9fd2-860dbb539dfd_isrForCrosstalkSources_999140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cd1413d7-efcd-47da-9fd2-860dbb539dfd_isrForCrosstalkSources_999140_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:18 proc_lsst.multi:146 [INFO] found job 17096734 in provider escience_a40 proc_lsst.multi INFO: found job 17096734 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdd6e4470f6394967f64fbf137ec4b39 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/afebc454-4b3e-4a42-97dc-f5686de856c7_isrForCrosstalkSources_999141_34.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/afebc454-4b3e-4a42-97dc-f5686de856c7_isrForCrosstalkSources_999141_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:19 proc_lsst.multi:146 [INFO] found job 17096735 in provider escience_a40 proc_lsst.multi INFO: found job 17096735 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 021d0cea6258cdd23c18eb0873357d88 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/b9a4cd01-62c4-48f5-a394-f1604031d935_isrForCrosstalkSources_999139_16.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/b9a4cd01-62c4-48f5-a394-f1604031d935_isrForCrosstalkSources_999139_16.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a387dcd44e5cb4ce6f27fd276419158c with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2eef36bc-3953-4229-9a3e-4de3c43a5fb3_isrForCrosstalkSources_999141_41.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/2eef36bc-3953-4229-9a3e-4de3c43a5fb3_isrForCrosstalkSources_999141_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:19 proc_lsst.multi:146 [INFO] found job 17096736 in provider escience_a40 proc_lsst.multi INFO: found job 17096736 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry decba366c7dab214a109fb5de7646172 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5d813990-1b33-424b-ac28-ed8de4059297_isrForCrosstalkSources_999139_10.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5d813990-1b33-424b-ac28-ed8de4059297_isrForCrosstalkSources_999139_10.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac9f715799d3e6ca962189762bfd5805 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/987d404d-ed01-41f0-b48a-11696c787166_isrForCrosstalkSources_999139_5.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/987d404d-ed01-41f0-b48a-11696c787166_isrForCrosstalkSources_999139_5.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b820018cf96a8761429ae7f3fd1377d with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/d8535222-069c-482e-b19c-6f3de79a4567_isrForCrosstalkSources_999141_14.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/d8535222-069c-482e-b19c-6f3de79a4567_isrForCrosstalkSources_999141_14.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47ac259b73926277a534e9a5d00c9a6c with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/24926554-8d84-4b79-80fb-6f5d85cd1f3e_isrForCrosstalkSources_999141_51.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/24926554-8d84-4b79-80fb-6f5d85cd1f3e_isrForCrosstalkSources_999141_51.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d406c9d7413570d4815418e5776c4fd1 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/36a114e8-bf5e-4e36-bef1-8059285e5751_isrForCrosstalkSources_999141_23.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/36a114e8-bf5e-4e36-bef1-8059285e5751_isrForCrosstalkSources_999141_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:20 proc_lsst.multi:146 [INFO] found job 17096737 in provider escience_a40 proc_lsst.multi INFO: found job 17096737 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3ad131548f46151ce6f12b464bb4174 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/4a995351-da31-473f-99ee-2e4409780701_isrForCrosstalkSources_999141_13.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/4a995351-da31-473f-99ee-2e4409780701_isrForCrosstalkSources_999141_13.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 354f1d1bae18875a489a78c59a5cbc53 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/fe19ee28-829e-4b91-af0d-0cb40330745d_isrForCrosstalkSources_999139_39.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/fe19ee28-829e-4b91-af0d-0cb40330745d_isrForCrosstalkSources_999139_39.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e1dddef279d82f9692fa66617ba5740 with result from task 120 parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:21 proc_lsst.multi:146 [INFO] found job 17096738 in provider escience_a40 proc_lsst.multi INFO: found job 17096738 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/8916f19b-9f60-42c1-983b-25eb8e23a0fd_isrForCrosstalkSources_999141_29.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/8916f19b-9f60-42c1-983b-25eb8e23a0fd_isrForCrosstalkSources_999141_29.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50ae189c1329dd5d016ae19c552f0b78 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4761dde5-5f5a-4886-8172-12f525336ae9_isrForCrosstalkSources_999139_32.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4761dde5-5f5a-4886-8172-12f525336ae9_isrForCrosstalkSources_999139_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:21 proc_lsst.multi:146 [INFO] found job 17096739 in provider escience_a40 proc_lsst.multi INFO: found job 17096739 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6d542995f6c4990e58ff15ba228c04b with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/70be6fe1-8a45-4eff-a076-524122c8771a_isrForCrosstalkSources_999141_61.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/70be6fe1-8a45-4eff-a076-524122c8771a_isrForCrosstalkSources_999141_61.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab2078876788947a18166010e6146c28 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/0540d748-cd91-4a3a-a4ae-bca5b0e3e770_isrForCrosstalkSources_999140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/0540d748-cd91-4a3a-a4ae-bca5b0e3e770_isrForCrosstalkSources_999140_5.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f684d94e9c667a11e22132127ccbd48 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9bd3b7b4-1e08-442b-b3a6-c9c1aedb2993_isrForCrosstalkSources_999140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/9bd3b7b4-1e08-442b-b3a6-c9c1aedb2993_isrForCrosstalkSources_999140_57.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c73f747e3a215f2dc9b49e22c1fee8c0 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b9330f32-c7e5-4b05-be46-3b36e18fe6be_isrForCrosstalkSources_999140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b9330f32-c7e5-4b05-be46-3b36e18fe6be_isrForCrosstalkSources_999140_1.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f09c91c40fb7351af9323da669978bf with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e1d940c3-f11b-4eda-8158-778200594811_isrForCrosstalkSources_999140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e1d940c3-f11b-4eda-8158-778200594811_isrForCrosstalkSources_999140_24.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a83a1efa26900c252eac33e150d3d2d with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/05e0841c-9d45-424c-82e7-88b982bfcebd_isrForCrosstalkSources_999139_27.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/05e0841c-9d45-424c-82e7-88b982bfcebd_isrForCrosstalkSources_999139_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 65 parsl.dataflow.strategy DEBUG: Executor multi has 65 active tasks, 46/0 running/pending blocks, and 47 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22674326706576 2024-03-16 00:04:22 proc_lsst.multi:146 [INFO] found job 13819 in provider local proc_lsst.multi INFO: found job 13819 in provider local 2024-03-16 00:04:22 proc_lsst.multi:146 [INFO] found job 17096695 in provider astro proc_lsst.multi INFO: found job 17096695 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc8245ce2a22996fc7f511f09fa2dfa0 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/688ca4cd-eee0-47c6-a919-948658526ecb_isrForCrosstalkSources_999139_56.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/688ca4cd-eee0-47c6-a919-948658526ecb_isrForCrosstalkSources_999139_56.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbf4172ef2cc9dfbc50cf44e10e17d75 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/e557be45-33d1-4f2a-bb84-4533ffb5fd8b_isrForCrosstalkSources_999139_6.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/e557be45-33d1-4f2a-bb84-4533ffb5fd8b_isrForCrosstalkSources_999139_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:22 proc_lsst.multi:146 [INFO] found job 17096696 in provider astro proc_lsst.multi INFO: found job 17096696 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d2b97a3ca5ae5dcd837058486108a18 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/4eb81f9a-9436-461d-9f8a-38eacfeb0443_isrForCrosstalkSources_999141_19.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/4eb81f9a-9436-461d-9f8a-38eacfeb0443_isrForCrosstalkSources_999141_19.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f46a8dd91c3943dd7c498b72958d0d7 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ef69143e-0545-4c04-85f7-75c2447ba6ca_isrForCrosstalkSources_999141_62.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ef69143e-0545-4c04-85f7-75c2447ba6ca_isrForCrosstalkSources_999141_62.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40f12bd6c18945bcd3b0fe706994f124 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/854e2e01-61c7-4efd-94c4-eb8eed30b932_isrForCrosstalkSources_999141_54.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/854e2e01-61c7-4efd-94c4-eb8eed30b932_isrForCrosstalkSources_999141_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:23 proc_lsst.multi:146 [INFO] found job 17096697 in provider astro proc_lsst.multi INFO: found job 17096697 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e80373c163c36746e3162a0a57eb0c16 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/eb517a6a-ee0c-4666-892f-30e88be51ddb_isrForCrosstalkSources_999139_17.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/eb517a6a-ee0c-4666-892f-30e88be51ddb_isrForCrosstalkSources_999139_17.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 829df48e864e9c3c4aceea5688ff0eaa with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/49ef8480-35f3-4fc0-b9f2-af507ccfdb2c_isrForCrosstalkSources_999141_21.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/49ef8480-35f3-4fc0-b9f2-af507ccfdb2c_isrForCrosstalkSources_999141_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:23 proc_lsst.multi:146 [INFO] found job 17096698 in provider astro proc_lsst.multi INFO: found job 17096698 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:24 proc_lsst.multi:146 [INFO] found job 17096699 in provider astro proc_lsst.multi INFO: found job 17096699 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:24 proc_lsst.multi:146 [INFO] found job 17096700 in provider astro proc_lsst.multi INFO: found job 17096700 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:25 proc_lsst.multi:146 [INFO] found job 17096701 in provider astro proc_lsst.multi INFO: found job 17096701 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:25 proc_lsst.multi:146 [INFO] found job 17096702 in provider astro proc_lsst.multi INFO: found job 17096702 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:26 proc_lsst.multi:146 [INFO] found job 17096703 in provider astro proc_lsst.multi INFO: found job 17096703 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b746ab3b54efc89bbc075e98d513a13a with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/ff2281fa-489e-45e1-9d9c-b43db3718203_isrForCrosstalkSources_999140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/ff2281fa-489e-45e1-9d9c-b43db3718203_isrForCrosstalkSources_999140_32.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf718e5215c49ceed384a1333004de8f with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/b6dff8a5-3177-4103-8ec0-f78da015f308_isrForCrosstalkSources_999139_38.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/b6dff8a5-3177-4103-8ec0-f78da015f308_isrForCrosstalkSources_999139_38.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84a63994bfc550c1e178493b016dd792 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/5f8806ce-e91f-4e6d-b599-720290235fd3_isrForCrosstalkSources_999140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/5f8806ce-e91f-4e6d-b599-720290235fd3_isrForCrosstalkSources_999140_51.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a10d4d57e2957e533ee2cf2062e0013 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/b310f608-77d5-4bbf-ad1a-8bb586e52b25_isrForCrosstalkSources_999141_55.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/b310f608-77d5-4bbf-ad1a-8bb586e52b25_isrForCrosstalkSources_999141_55.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7592b2d35fe1e7ee9837228585505de6 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/159f3402-4837-4527-a533-a3964ad9f61f_isrForCrosstalkSources_999139_44.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/159f3402-4837-4527-a533-a3964ad9f61f_isrForCrosstalkSources_999139_44.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a6e405e9ecbb6fecdc27b53fcac229b with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/5c77d690-6cb9-4fdd-8862-65ee2d425690_isrForCrosstalkSources_999140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/5c77d690-6cb9-4fdd-8862-65ee2d425690_isrForCrosstalkSources_999140_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:27 proc_lsst.multi:146 [INFO] found job 17096704 in provider astro proc_lsst.multi INFO: found job 17096704 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f493cb85cffedbef480ecef37f3b280a with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/3874738b-22e7-42b8-a186-e9d1e96c3595_isrForCrosstalkSources_999141_3.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/3874738b-22e7-42b8-a186-e9d1e96c3595_isrForCrosstalkSources_999141_3.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2485a4df25651824b231074822c3437c with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/0cfc68a0-f7c3-404a-bd55-e0e860d26938_isrForCrosstalkSources_999139_3.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/0cfc68a0-f7c3-404a-bd55-e0e860d26938_isrForCrosstalkSources_999139_3.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7732345aa46626fe2335c90fb6873ab0 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/43a4ab38-25f0-4a7f-874e-9263f0bc3960_isrForCrosstalkSources_999141_7.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/43a4ab38-25f0-4a7f-874e-9263f0bc3960_isrForCrosstalkSources_999141_7.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92084fea8f88d97e8a75c0ad6cbd8c20 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/da192a09-d482-406e-aae0-b3c1931b7bbf_isrForCrosstalkSources_999139_51.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/da192a09-d482-406e-aae0-b3c1931b7bbf_isrForCrosstalkSources_999139_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:27 proc_lsst.multi:146 [INFO] found job 17096705 in provider astro proc_lsst.multi INFO: found job 17096705 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddbdd8f367f6b0a34bff2cbfceaa3c48 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a6586ecf-be9a-4d1b-b3b2-dc1dbe9adc47_isrForCrosstalkSources_999141_4.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/a6586ecf-be9a-4d1b-b3b2-dc1dbe9adc47_isrForCrosstalkSources_999141_4.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 433e38f1c297f045fe1be9800ef6c331 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b6b066f1-08c1-4f16-999e-2bf8f20aa490_isrForCrosstalkSources_999140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b6b066f1-08c1-4f16-999e-2bf8f20aa490_isrForCrosstalkSources_999140_7.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14ca08c52a16871172c887fc2b25eabd with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4865bd85-a860-4e8a-8242-adb3446a68fa_isrForCrosstalkSources_999140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/4865bd85-a860-4e8a-8242-adb3446a68fa_isrForCrosstalkSources_999140_49.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cf8b62452f9ba55ceef17542477f356 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a90c719a-4a51-49b1-9bed-f7359d7f71bf_isrForCrosstalkSources_999140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/a90c719a-4a51-49b1-9bed-f7359d7f71bf_isrForCrosstalkSources_999140_18.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16ac508832d7268244ebde3662a6df70 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cffd2098-0880-45fc-a25f-972605e94f36_isrForCrosstalkSources_999140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cffd2098-0880-45fc-a25f-972605e94f36_isrForCrosstalkSources_999140_15.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8226cdc575b78356fb5365d2550e41c with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cbf24905-1b5c-4de3-bd61-d70ef3e61846_isrForCrosstalkSources_999140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cbf24905-1b5c-4de3-bd61-d70ef3e61846_isrForCrosstalkSources_999140_58.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 998b7590b78f8d557cc4eb152d8680a7 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/7d0b7a61-c089-409c-8ce1-f7e911a2d897_isrForCrosstalkSources_999141_58.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/7d0b7a61-c089-409c-8ce1-f7e911a2d897_isrForCrosstalkSources_999141_58.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adb27294db730505bf9d1f3bdb8872d3 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7d727e03-6c7e-4d0a-bddf-2033eab7e70b_isrForCrosstalkSources_999139_46.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/7d727e03-6c7e-4d0a-bddf-2033eab7e70b_isrForCrosstalkSources_999139_46.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1ed7613dfa6c938227891c771b9599b with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/df48b1e4-d0c3-46f0-b79e-bd7015a595d8_isrForCrosstalkSources_999141_42.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/df48b1e4-d0c3-46f0-b79e-bd7015a595d8_isrForCrosstalkSources_999141_42.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a80fe4bf04e8b8c5f8e5b0e866c8dc6 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/847e8b4a-19ab-44dc-811f-6a0adeef9877_isrForCrosstalkSources_999139_41.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/847e8b4a-19ab-44dc-811f-6a0adeef9877_isrForCrosstalkSources_999139_41.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c298e53d313d263138920defd10a8fa with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/42772196-1b05-4251-bf8e-1b4125694504_isrForCrosstalkSources_999141_53.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/42772196-1b05-4251-bf8e-1b4125694504_isrForCrosstalkSources_999141_53.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88d4e8f90512810e0879f0f4b601a38a with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e1f4f340-b147-4d71-9b34-a761f79377bd_isrForCrosstalkSources_999140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e1f4f340-b147-4d71-9b34-a761f79377bd_isrForCrosstalkSources_999140_35.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed9696c276d20d910edb8c8de6cc8f48 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/2dbb05d8-3c16-4c79-92bb-7b6e69c759c2_isrForCrosstalkSources_999140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/2dbb05d8-3c16-4c79-92bb-7b6e69c759c2_isrForCrosstalkSources_999140_36.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61d53507b3735d8ef1a3352b8ad57b3b with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/1fdfba6f-a689-4c09-95e0-d4c2c2e77916_isrForCrosstalkSources_999141_40.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/1fdfba6f-a689-4c09-95e0-d4c2c2e77916_isrForCrosstalkSources_999141_40.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e1365bb81f51bd8a4766112eef5498c with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/71b048a4-91ca-4070-9dbd-a96ec0a04202_isrForCrosstalkSources_999141_44.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/71b048a4-91ca-4070-9dbd-a96ec0a04202_isrForCrosstalkSources_999141_44.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c586f1b5cbd877506bcc6dbf935707ee with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/710c5390-5648-40de-9c92-5e90dab29c4d_isrForCrosstalkSources_999139_48.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/710c5390-5648-40de-9c92-5e90dab29c4d_isrForCrosstalkSources_999139_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:28 proc_lsst.multi:146 [INFO] found job 17096706 in provider astro proc_lsst.multi INFO: found job 17096706 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4730828a9aca0f6ecadca46541a390c with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/0b61a90d-c04f-414e-8481-506099ae5f0f_isrForCrosstalkSources_999140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/0b61a90d-c04f-414e-8481-506099ae5f0f_isrForCrosstalkSources_999140_37.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f805ae214e249449eb8abee0b4d3b00 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/bc9a4eb6-26fb-4ab1-9c93-a1717e4c65cf_isrForCrosstalkSources_999139_59.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/bc9a4eb6-26fb-4ab1-9c93-a1717e4c65cf_isrForCrosstalkSources_999139_59.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac1cbe4461007f18a132e9bd9730b2a9 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/da27ad95-035e-4511-973c-9cfa8de0cee0_isrForCrosstalkSources_999139_55.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/da27ad95-035e-4511-973c-9cfa8de0cee0_isrForCrosstalkSources_999139_55.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c4854461261c241956439598f06890e with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fd88e0cd-d10c-41c8-a818-100f7b69d508_isrForCrosstalkSources_999141_33.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fd88e0cd-d10c-41c8-a818-100f7b69d508_isrForCrosstalkSources_999141_33.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a6bcd5dfb6531864131ed5ee1a4ad18 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/cedb1ac5-3a7d-4b0a-80e6-bea2ad1be527_isrForCrosstalkSources_999139_8.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/cedb1ac5-3a7d-4b0a-80e6-bea2ad1be527_isrForCrosstalkSources_999139_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:29 proc_lsst.multi:146 [INFO] found job 17096707 in provider astro proc_lsst.multi INFO: found job 17096707 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3bedddb791356068d81b40b8cb17130 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/021d1d95-130c-4e24-8c8d-e5478f5cd68c_isrForCrosstalkSources_999139_26.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/021d1d95-130c-4e24-8c8d-e5478f5cd68c_isrForCrosstalkSources_999139_26.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cd2e3793ac74e8c794bcf782020caa2 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cb9df948-ae18-42df-a0b8-247ce43d7888_isrForCrosstalkSources_999140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/cb9df948-ae18-42df-a0b8-247ce43d7888_isrForCrosstalkSources_999140_9.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a643f376f0b7881405ba28f47e68d47b with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ef22d0bf-a6bd-4320-88ce-85cd247d387a_isrForCrosstalkSources_999141_46.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/ef22d0bf-a6bd-4320-88ce-85cd247d387a_isrForCrosstalkSources_999141_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:30 proc_lsst.multi:146 [INFO] found job 17096708 in provider astro proc_lsst.multi INFO: found job 17096708 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56db3e9040a4f6e2b1c632fef960bfc5 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b5cdcc0e-90a2-4bc7-a659-21034a127550_isrForCrosstalkSources_999140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b5cdcc0e-90a2-4bc7-a659-21034a127550_isrForCrosstalkSources_999140_42.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6288e5dd2894faf923a74cca3ee286e8 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1c0027bb-83a6-4d22-b84c-63ff8d5eeeb4_isrForCrosstalkSources_999139_13.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/1c0027bb-83a6-4d22-b84c-63ff8d5eeeb4_isrForCrosstalkSources_999139_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:31 proc_lsst.multi:146 [INFO] found job 17096709 in provider astro proc_lsst.multi INFO: found job 17096709 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2103f13326e8c4b80f582a1ed84b4450 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/cb084fea-c526-46ec-93be-565170ef9e13_isrForCrosstalkSources_999141_52.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/cb084fea-c526-46ec-93be-565170ef9e13_isrForCrosstalkSources_999141_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:31 proc_lsst.multi:146 [INFO] found job 17096710 in provider astro proc_lsst.multi INFO: found job 17096710 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:31 proc_lsst.multi:146 [INFO] found job 17096711 in provider astro proc_lsst.multi INFO: found job 17096711 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12bce6b9f8d96e10e8329ff78516fff8 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/63c97003-2994-443d-9856-b051fced3fbb_isrForCrosstalkSources_999140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/63c97003-2994-443d-9856-b051fced3fbb_isrForCrosstalkSources_999140_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:32 proc_lsst.multi:146 [INFO] found job 17096712 in provider astro proc_lsst.multi INFO: found job 17096712 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03483d768203035c0569940f2f9fd4f5 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e093bfa0-1d54-44fe-8f19-047d4724718e_isrForCrosstalkSources_999140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/e093bfa0-1d54-44fe-8f19-047d4724718e_isrForCrosstalkSources_999140_34.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b0248207a69133844092ae09fc74e29 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c4b1e4d1-2abb-4180-bb6b-bd954d0b098b_isrForCrosstalkSources_999140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c4b1e4d1-2abb-4180-bb6b-bd954d0b098b_isrForCrosstalkSources_999140_38.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a4746f358a0ad6137ee70a4108ef0df with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/d339dac6-95e9-4af5-9a37-4df036cd71dc_isrForCrosstalkSources_999140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/d339dac6-95e9-4af5-9a37-4df036cd71dc_isrForCrosstalkSources_999140_43.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0fa624649b76e36b06c12f233320258 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/28c61392-0104-4ff5-9e2d-15786f31e012_isrForCrosstalkSources_999140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/28c61392-0104-4ff5-9e2d-15786f31e012_isrForCrosstalkSources_999140_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:32 proc_lsst.multi:146 [INFO] found job 17096713 in provider astro proc_lsst.multi INFO: found job 17096713 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff24e4deeca24b12cdbb163feee9c8f0 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/308d8920-6d8f-4980-8cf5-9a63255de9b7_isrForCrosstalkSources_999141_59.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/308d8920-6d8f-4980-8cf5-9a63255de9b7_isrForCrosstalkSources_999141_59.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 371ca17966531e14ed26ba2ec1310a05 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/be0a56da-1882-4750-a268-d17ece32a71c_isrForCrosstalkSources_999141_25.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/be0a56da-1882-4750-a268-d17ece32a71c_isrForCrosstalkSources_999141_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:32 proc_lsst.multi:146 [INFO] found job 17096714 in provider astro proc_lsst.multi INFO: found job 17096714 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e204217d5b2d420c9993f1465292dfce with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/de9d7964-c05c-477e-aaab-7929eb1991a4_isrForCrosstalkSources_999141_5.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/de9d7964-c05c-477e-aaab-7929eb1991a4_isrForCrosstalkSources_999141_5.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92d49d549f1153997679027c5367a60b with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/da1f6e40-ac56-4b33-bdf3-3c5c861c5ffc_isrForCrosstalkSources_999140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/da1f6e40-ac56-4b33-bdf3-3c5c861c5ffc_isrForCrosstalkSources_999140_22.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fe3e69238eb6b71d2790017325f7fd4 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e3f9664f-cd0a-494d-8317-685911615730_isrForCrosstalkSources_999141_11.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/e3f9664f-cd0a-494d-8317-685911615730_isrForCrosstalkSources_999141_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:32 proc_lsst.multi:146 [INFO] found job 17096715 in provider astro proc_lsst.multi INFO: found job 17096715 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:33 proc_lsst.multi:146 [INFO] found job 17096716 in provider astro proc_lsst.multi INFO: found job 17096716 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:33 proc_lsst.multi:146 [INFO] found job 17096717 in provider astro proc_lsst.multi INFO: found job 17096717 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:33 proc_lsst.multi:146 [INFO] found job 17096718 in provider astro proc_lsst.multi INFO: found job 17096718 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:33 proc_lsst.multi:146 [INFO] found job 17096719 in provider astro proc_lsst.multi INFO: found job 17096719 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:33 proc_lsst.multi:146 [INFO] found job 17096720 in provider astro proc_lsst.multi INFO: found job 17096720 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:34 proc_lsst.multi:146 [INFO] found job 17096721 in provider astro proc_lsst.multi INFO: found job 17096721 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:34 proc_lsst.multi:146 [INFO] found job 17096722 in provider astro proc_lsst.multi INFO: found job 17096722 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:34 proc_lsst.multi:146 [INFO] found job 17096723 in provider astro proc_lsst.multi INFO: found job 17096723 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:34 proc_lsst.multi:146 [INFO] found job 17096724 in provider astro proc_lsst.multi INFO: found job 17096724 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096695,17096696,17096697,17096698,17096699,17096700,17096701,17096702,17096703,17096704,17096705,17096706,17096707,17096708,17096709,17096710,17096711,17096712,17096713,17096714,17096715,17096716,17096717,17096718,17096719,17096720,17096721,17096722,17096723,17096724' parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e114b5080de81a3fa6bc5b44235173a with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c07ff830-bcb1-447e-a492-446b761aa7cc_isrForCrosstalkSources_999140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/c07ff830-bcb1-447e-a492-446b761aa7cc_isrForCrosstalkSources_999140_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096722 R 17096723 R 17096724 R 17096714 R 17096715 R 17096716 R 17096717 R 17096718 R 17096719 R 17096720 R 17096721 R 17096703 R 17096704 R 17096705 R 17096706 R 17096707 R 17096708 R 17096709 R 17096710 R 17096711 R 17096712 R 17096713 R 17096696 R 17096697 R 17096698 R 17096699 R 17096700 R 17096701 R 17096702 R 17096695 R parsl.providers.slurm.slurm DEBUG: Updating job 17096722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096695 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:34 proc_lsst.multi:146 [INFO] found job 17096725 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096725 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:35 proc_lsst.multi:146 [INFO] found job 17096726 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096726 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fa33a5aaeda14244a128cf364809d03 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/c4d00cff-512b-409b-9fbb-1fc46ab41aec_isrForCrosstalkSources_999141_17.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/c4d00cff-512b-409b-9fbb-1fc46ab41aec_isrForCrosstalkSources_999141_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:35 proc_lsst.multi:146 [INFO] found job 17096727 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096727 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:35 proc_lsst.multi:146 [INFO] found job 17096728 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096728 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adf5f9ffc3ed8cdd1a969ab652b9c593 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fed34728-00a3-4b15-be69-bb7624050711_isrForCrosstalkSources_999141_30.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/fed34728-00a3-4b15-be69-bb7624050711_isrForCrosstalkSources_999141_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:35 proc_lsst.multi:146 [INFO] found job 17096729 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096729 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096725,17096726,17096727,17096728,17096729' parsl.providers.slurm.slurm DEBUG: squeue returned 17096728 R 17096729 R 17096725 R 17096726 R 17096727 R parsl.providers.slurm.slurm DEBUG: Updating job 17096728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096727 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:36 proc_lsst.multi:146 [INFO] found job 17096730 in provider escience_a40 proc_lsst.multi INFO: found job 17096730 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:36 proc_lsst.multi:146 [INFO] found job 17096731 in provider escience_a40 proc_lsst.multi INFO: found job 17096731 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:36 proc_lsst.multi:146 [INFO] found job 17096732 in provider escience_a40 proc_lsst.multi INFO: found job 17096732 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c4b87bb280b485476af4ee473472599 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f0c8b8ca-0d96-4715-b6ea-103c559db310_isrForCrosstalkSources_999141_32.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f0c8b8ca-0d96-4715-b6ea-103c559db310_isrForCrosstalkSources_999141_32.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dc5705d086d6116ae16837dced74ca9 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f782a802-c524-4b1d-8b44-b6b770b40cb4_isrForCrosstalkSources_999141_6.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/f782a802-c524-4b1d-8b44-b6b770b40cb4_isrForCrosstalkSources_999141_6.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 306aaedaabe3fae9efcc2619dba7f8f4 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/50dfc65c-8dc8-4a10-926f-7d41c24fa557_isrForCrosstalkSources_999141_43.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999141/50dfc65c-8dc8-4a10-926f-7d41c24fa557_isrForCrosstalkSources_999141_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:36 proc_lsst.multi:146 [INFO] found job 17096733 in provider escience_a40 proc_lsst.multi INFO: found job 17096733 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3594404e6589cf5e639edabc35b625d with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5010511d-0f0e-49b2-aaf0-51ece2c01c36_isrForCrosstalkSources_999139_60.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/5010511d-0f0e-49b2-aaf0-51ece2c01c36_isrForCrosstalkSources_999139_60.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0190fb4d28bfd868c9159af1a6161f5e with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4bd292cf-bfb4-407e-b140-63fb2c7c03d9_isrForCrosstalkSources_999139_58.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/4bd292cf-bfb4-407e-b140-63fb2c7c03d9_isrForCrosstalkSources_999139_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:36 proc_lsst.multi:146 [INFO] found job 17096734 in provider escience_a40 proc_lsst.multi INFO: found job 17096734 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af9d3b34bc160dab0c125e3c476f4a2e with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b76fd1a9-1517-419e-a654-b675456978d9_isrForCrosstalkSources_999140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/b76fd1a9-1517-419e-a654-b675456978d9_isrForCrosstalkSources_999140_12.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef22a5fb1503e638cc206e8102a78829 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/890499ac-419f-4739-86fa-be6d7d62900f_isrForCrosstalkSources_999139_11.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999139/890499ac-419f-4739-86fa-be6d7d62900f_isrForCrosstalkSources_999139_11.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d302be1d536891872d2169d482a54d5 with result from task 178 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 186 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/8b82ffb6-7649-4fbb-b1a8-adb6e133e5ed_isrForCrosstalkSources_999140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/submit/DEEP/20210518/A1e/science#step0/20240316T000255Z/logs/isrForCrosstalkSources/999140/8b82ffb6-7649-4fbb-b1a8-adb6e133e5ed_isrForCrosstalkSources_999140_61.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:37 proc_lsst.multi:146 [INFO] found job 17096735 in provider escience_a40 proc_lsst.multi INFO: found job 17096735 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:37 proc_lsst.multi:146 [INFO] found job 17096736 in provider escience_a40 proc_lsst.multi INFO: found job 17096736 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:37 proc_lsst.multi:146 [INFO] found job 17096737 in provider escience_a40 proc_lsst.multi INFO: found job 17096737 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:37 proc_lsst.multi:146 [INFO] found job 17096738 in provider escience_a40 proc_lsst.multi INFO: found job 17096738 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:04:38 proc_lsst.multi:146 [INFO] found job 17096739 in provider escience_a40 proc_lsst.multi INFO: found job 17096739 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096730,17096731,17096732,17096733,17096734,17096735,17096736,17096737,17096738,17096739' parsl.providers.slurm.slurm DEBUG: squeue returned 17096735 R 17096736 R 17096737 R 17096738 R 17096739 R 17096732 R 17096733 R 17096734 R 17096730 R 17096731 R parsl.providers.slurm.slurm DEBUG: Updating job 17096735 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096732 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096733 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096734 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096731 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 46/0 running/pending blocks, and 47 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (46) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 3.5762786865234375e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22674326706576 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=46, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 46 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 843b779dd9eb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 843b779dd9eb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4806ff9aba7c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4806ff9aba7c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ec03dd70d389 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ec03dd70d389 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3dd45de86a75 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3dd45de86a75 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6c30a3acfcf3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6c30a3acfcf3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c395c82e2247 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c395c82e2247 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2df544af5011 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2df544af5011 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c0e823698a2d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c0e823698a2d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 822c6cf2febd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 822c6cf2febd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bbbaae813144 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bbbaae813144 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4ac49c0e0f21 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4ac49c0e0f21 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c7c7e5746a7f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c7c7e5746a7f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0557677fc23d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0557677fc23d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e7bba159f33 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e7bba159f33 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d8bc27618bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d8bc27618bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 844af2b82f51 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 844af2b82f51 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ad2d4fc833bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ad2d4fc833bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a349285e0034 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a349285e0034 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5db66ecad360 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5db66ecad360 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9831be52aa48 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9831be52aa48 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d83016d091d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d83016d091d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 700166e64f8a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 700166e64f8a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 73ec38ac3a89 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 73ec38ac3a89 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4633c2355878 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4633c2355878 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: effdb3159771 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: effdb3159771 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e3763bd3f7f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e3763bd3f7f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce23b40f512d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce23b40f512d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: be764bcb6f19 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: be764bcb6f19 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4b20b94a680e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4b20b94a680e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0d857e42d2ac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0d857e42d2ac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7799d77a7195 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7799d77a7195 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e8e6d6724a63 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e8e6d6724a63 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c796ace50456 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c796ace50456 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 229b2941c06e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 229b2941c06e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ac844fd27e41 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ac844fd27e41 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1bdd104a8074 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1bdd104a8074 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4fc0228a0305 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4fc0228a0305 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e975771b2cfd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e975771b2cfd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 172d0582a0ce parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 172d0582a0ce parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 589e30f562de parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 589e30f562de parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 99aaa1d81302 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 99aaa1d81302 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c2d013b7b4dc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c2d013b7b4dc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2b1e138f5b97 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2b1e138f5b97 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 55336e56d32c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 55336e56d32c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 832740a46dad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 832740a46dad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 16af4270ab1f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 16af4270ab1f 2024-03-16 00:04:38 proc_lsst.multi:146 [INFO] found job 17096711 in provider astro proc_lsst.multi INFO: found job 17096711 in provider astro 2024-03-16 00:04:38 proc_lsst.multi:201 [INFO] cancelling 17096711 on provider astro proc_lsst.multi INFO: cancelling 17096711 on provider astro 2024-03-16 00:04:38 proc_lsst.multi:146 [INFO] found job 17096717 in provider astro proc_lsst.multi INFO: found job 17096717 in provider astro 2024-03-16 00:04:38 proc_lsst.multi:201 [INFO] cancelling 17096717 on provider astro proc_lsst.multi INFO: cancelling 17096717 on provider astro 2024-03-16 00:04:38 proc_lsst.multi:146 [INFO] found job 17096708 in provider astro proc_lsst.multi INFO: found job 17096708 in provider astro 2024-03-16 00:04:38 proc_lsst.multi:201 [INFO] cancelling 17096708 on provider astro proc_lsst.multi INFO: cancelling 17096708 on provider astro 2024-03-16 00:04:38 proc_lsst.multi:146 [INFO] found job 17096715 in provider astro proc_lsst.multi INFO: found job 17096715 in provider astro 2024-03-16 00:04:38 proc_lsst.multi:201 [INFO] cancelling 17096715 on provider astro proc_lsst.multi INFO: cancelling 17096715 on provider astro 2024-03-16 00:04:39 proc_lsst.multi:146 [INFO] found job 17096710 in provider astro proc_lsst.multi INFO: found job 17096710 in provider astro 2024-03-16 00:04:39 proc_lsst.multi:201 [INFO] cancelling 17096710 on provider astro proc_lsst.multi INFO: cancelling 17096710 on provider astro 2024-03-16 00:04:39 proc_lsst.multi:146 [INFO] found job 17096720 in provider astro proc_lsst.multi INFO: found job 17096720 in provider astro 2024-03-16 00:04:39 proc_lsst.multi:201 [INFO] cancelling 17096720 on provider astro proc_lsst.multi INFO: cancelling 17096720 on provider astro 2024-03-16 00:04:39 proc_lsst.multi:146 [INFO] found job 17096713 in provider astro proc_lsst.multi INFO: found job 17096713 in provider astro 2024-03-16 00:04:39 proc_lsst.multi:201 [INFO] cancelling 17096713 on provider astro proc_lsst.multi INFO: cancelling 17096713 on provider astro 2024-03-16 00:04:39 proc_lsst.multi:146 [INFO] found job 17096712 in provider astro proc_lsst.multi INFO: found job 17096712 in provider astro 2024-03-16 00:04:39 proc_lsst.multi:201 [INFO] cancelling 17096712 on provider astro proc_lsst.multi INFO: cancelling 17096712 on provider astro 2024-03-16 00:04:41 proc_lsst.multi:146 [INFO] found job 17096728 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096728 in provider escience_rtx6k 2024-03-16 00:04:41 proc_lsst.multi:201 [INFO] cancelling 17096728 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096728 on provider escience_rtx6k 2024-03-16 00:04:41 proc_lsst.multi:146 [INFO] found job 13819 in provider local proc_lsst.multi INFO: found job 13819 in provider local 2024-03-16 00:04:41 proc_lsst.multi:201 [INFO] cancelling 13819 on provider local proc_lsst.multi INFO: cancelling 13819 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 13819 2024-03-16 00:04:41 proc_lsst.multi:146 [INFO] found job 17096736 in provider escience_a40 proc_lsst.multi INFO: found job 17096736 in provider escience_a40 2024-03-16 00:04:41 proc_lsst.multi:201 [INFO] cancelling 17096736 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096736 on provider escience_a40 2024-03-16 00:04:41 proc_lsst.multi:146 [INFO] found job 17096702 in provider astro proc_lsst.multi INFO: found job 17096702 in provider astro 2024-03-16 00:04:41 proc_lsst.multi:201 [INFO] cancelling 17096702 on provider astro proc_lsst.multi INFO: cancelling 17096702 on provider astro 2024-03-16 00:04:41 proc_lsst.multi:146 [INFO] found job 17096701 in provider astro proc_lsst.multi INFO: found job 17096701 in provider astro 2024-03-16 00:04:41 proc_lsst.multi:201 [INFO] cancelling 17096701 on provider astro proc_lsst.multi INFO: cancelling 17096701 on provider astro 2024-03-16 00:04:41 proc_lsst.multi:146 [INFO] found job 17096698 in provider astro proc_lsst.multi INFO: found job 17096698 in provider astro 2024-03-16 00:04:41 proc_lsst.multi:201 [INFO] cancelling 17096698 on provider astro proc_lsst.multi INFO: cancelling 17096698 on provider astro 2024-03-16 00:04:41 proc_lsst.multi:146 [INFO] found job 17096695 in provider astro proc_lsst.multi INFO: found job 17096695 in provider astro 2024-03-16 00:04:41 proc_lsst.multi:201 [INFO] cancelling 17096695 on provider astro proc_lsst.multi INFO: cancelling 17096695 on provider astro 2024-03-16 00:04:42 proc_lsst.multi:146 [INFO] found job 17096697 in provider astro proc_lsst.multi INFO: found job 17096697 in provider astro 2024-03-16 00:04:42 proc_lsst.multi:201 [INFO] cancelling 17096697 on provider astro proc_lsst.multi INFO: cancelling 17096697 on provider astro 2024-03-16 00:04:42 proc_lsst.multi:146 [INFO] found job 17096700 in provider astro proc_lsst.multi INFO: found job 17096700 in provider astro 2024-03-16 00:04:42 proc_lsst.multi:201 [INFO] cancelling 17096700 on provider astro proc_lsst.multi INFO: cancelling 17096700 on provider astro 2024-03-16 00:04:42 proc_lsst.multi:146 [INFO] found job 17096699 in provider astro proc_lsst.multi INFO: found job 17096699 in provider astro 2024-03-16 00:04:42 proc_lsst.multi:201 [INFO] cancelling 17096699 on provider astro proc_lsst.multi INFO: cancelling 17096699 on provider astro 2024-03-16 00:04:42 proc_lsst.multi:146 [INFO] found job 17096704 in provider astro proc_lsst.multi INFO: found job 17096704 in provider astro 2024-03-16 00:04:42 proc_lsst.multi:201 [INFO] cancelling 17096704 on provider astro proc_lsst.multi INFO: cancelling 17096704 on provider astro 2024-03-16 00:04:42 proc_lsst.multi:146 [INFO] found job 17096703 in provider astro proc_lsst.multi INFO: found job 17096703 in provider astro 2024-03-16 00:04:42 proc_lsst.multi:201 [INFO] cancelling 17096703 on provider astro proc_lsst.multi INFO: cancelling 17096703 on provider astro 2024-03-16 00:04:42 proc_lsst.multi:146 [INFO] found job 17096696 in provider astro proc_lsst.multi INFO: found job 17096696 in provider astro 2024-03-16 00:04:42 proc_lsst.multi:201 [INFO] cancelling 17096696 on provider astro proc_lsst.multi INFO: cancelling 17096696 on provider astro 2024-03-16 00:04:43 proc_lsst.multi:146 [INFO] found job 17096724 in provider astro proc_lsst.multi INFO: found job 17096724 in provider astro 2024-03-16 00:04:43 proc_lsst.multi:201 [INFO] cancelling 17096724 on provider astro proc_lsst.multi INFO: cancelling 17096724 on provider astro 2024-03-16 00:04:43 proc_lsst.multi:146 [INFO] found job 17096738 in provider escience_a40 proc_lsst.multi INFO: found job 17096738 in provider escience_a40 2024-03-16 00:04:43 proc_lsst.multi:201 [INFO] cancelling 17096738 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096738 on provider escience_a40 2024-03-16 00:04:43 proc_lsst.multi:146 [INFO] found job 17096731 in provider escience_a40 proc_lsst.multi INFO: found job 17096731 in provider escience_a40 2024-03-16 00:04:43 proc_lsst.multi:201 [INFO] cancelling 17096731 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096731 on provider escience_a40 2024-03-16 00:04:43 proc_lsst.multi:146 [INFO] found job 17096721 in provider astro proc_lsst.multi INFO: found job 17096721 in provider astro 2024-03-16 00:04:43 proc_lsst.multi:201 [INFO] cancelling 17096721 on provider astro proc_lsst.multi INFO: cancelling 17096721 on provider astro 2024-03-16 00:04:43 proc_lsst.multi:146 [INFO] found job 17096722 in provider astro proc_lsst.multi INFO: found job 17096722 in provider astro 2024-03-16 00:04:43 proc_lsst.multi:201 [INFO] cancelling 17096722 on provider astro proc_lsst.multi INFO: cancelling 17096722 on provider astro 2024-03-16 00:04:43 proc_lsst.multi:146 [INFO] found job 17096723 in provider astro proc_lsst.multi INFO: found job 17096723 in provider astro 2024-03-16 00:04:43 proc_lsst.multi:201 [INFO] cancelling 17096723 on provider astro proc_lsst.multi INFO: cancelling 17096723 on provider astro 2024-03-16 00:04:44 proc_lsst.multi:146 [INFO] found job 17096716 in provider astro proc_lsst.multi INFO: found job 17096716 in provider astro 2024-03-16 00:04:44 proc_lsst.multi:201 [INFO] cancelling 17096716 on provider astro proc_lsst.multi INFO: cancelling 17096716 on provider astro 2024-03-16 00:04:44 proc_lsst.multi:146 [INFO] found job 17096739 in provider escience_a40 proc_lsst.multi INFO: found job 17096739 in provider escience_a40 2024-03-16 00:04:44 proc_lsst.multi:201 [INFO] cancelling 17096739 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096739 on provider escience_a40 2024-03-16 00:04:44 proc_lsst.multi:146 [INFO] found job 17096714 in provider astro proc_lsst.multi INFO: found job 17096714 in provider astro 2024-03-16 00:04:44 proc_lsst.multi:201 [INFO] cancelling 17096714 on provider astro proc_lsst.multi INFO: cancelling 17096714 on provider astro 2024-03-16 00:04:44 proc_lsst.multi:146 [INFO] found job 17096706 in provider astro proc_lsst.multi INFO: found job 17096706 in provider astro 2024-03-16 00:04:44 proc_lsst.multi:201 [INFO] cancelling 17096706 on provider astro proc_lsst.multi INFO: cancelling 17096706 on provider astro 2024-03-16 00:04:44 proc_lsst.multi:146 [INFO] found job 17096726 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096726 in provider escience_rtx6k 2024-03-16 00:04:44 proc_lsst.multi:201 [INFO] cancelling 17096726 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096726 on provider escience_rtx6k 2024-03-16 00:04:45 proc_lsst.multi:146 [INFO] found job 17096732 in provider escience_a40 proc_lsst.multi INFO: found job 17096732 in provider escience_a40 2024-03-16 00:04:45 proc_lsst.multi:201 [INFO] cancelling 17096732 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096732 on provider escience_a40 2024-03-16 00:04:45 proc_lsst.multi:146 [INFO] found job 17096727 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096727 in provider escience_rtx6k 2024-03-16 00:04:45 proc_lsst.multi:201 [INFO] cancelling 17096727 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096727 on provider escience_rtx6k 2024-03-16 00:04:45 proc_lsst.multi:146 [INFO] found job 17096735 in provider escience_a40 proc_lsst.multi INFO: found job 17096735 in provider escience_a40 2024-03-16 00:04:45 proc_lsst.multi:201 [INFO] cancelling 17096735 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096735 on provider escience_a40 2024-03-16 00:04:45 proc_lsst.multi:146 [INFO] found job 17096719 in provider astro proc_lsst.multi INFO: found job 17096719 in provider astro 2024-03-16 00:04:45 proc_lsst.multi:201 [INFO] cancelling 17096719 on provider astro proc_lsst.multi INFO: cancelling 17096719 on provider astro 2024-03-16 00:04:45 proc_lsst.multi:146 [INFO] found job 17096734 in provider escience_a40 proc_lsst.multi INFO: found job 17096734 in provider escience_a40 2024-03-16 00:04:45 proc_lsst.multi:201 [INFO] cancelling 17096734 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096734 on provider escience_a40 2024-03-16 00:04:45 proc_lsst.multi:146 [INFO] found job 17096718 in provider astro proc_lsst.multi INFO: found job 17096718 in provider astro 2024-03-16 00:04:45 proc_lsst.multi:201 [INFO] cancelling 17096718 on provider astro proc_lsst.multi INFO: cancelling 17096718 on provider astro 2024-03-16 00:04:46 proc_lsst.multi:146 [INFO] found job 17096730 in provider escience_a40 proc_lsst.multi INFO: found job 17096730 in provider escience_a40 2024-03-16 00:04:46 proc_lsst.multi:201 [INFO] cancelling 17096730 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096730 on provider escience_a40 2024-03-16 00:04:46 proc_lsst.multi:146 [INFO] found job 17096705 in provider astro proc_lsst.multi INFO: found job 17096705 in provider astro 2024-03-16 00:04:46 proc_lsst.multi:201 [INFO] cancelling 17096705 on provider astro proc_lsst.multi INFO: cancelling 17096705 on provider astro 2024-03-16 00:04:46 proc_lsst.multi:146 [INFO] found job 17096733 in provider escience_a40 proc_lsst.multi INFO: found job 17096733 in provider escience_a40 2024-03-16 00:04:46 proc_lsst.multi:201 [INFO] cancelling 17096733 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096733 on provider escience_a40 2024-03-16 00:04:46 proc_lsst.multi:146 [INFO] found job 17096737 in provider escience_a40 proc_lsst.multi INFO: found job 17096737 in provider escience_a40 2024-03-16 00:04:46 proc_lsst.multi:201 [INFO] cancelling 17096737 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096737 on provider escience_a40 2024-03-16 00:04:47 proc_lsst.multi:146 [INFO] found job 17096729 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096729 in provider escience_rtx6k 2024-03-16 00:04:47 proc_lsst.multi:201 [INFO] cancelling 17096729 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096729 on provider escience_rtx6k 2024-03-16 00:04:47 proc_lsst.multi:146 [INFO] found job 17096707 in provider astro proc_lsst.multi INFO: found job 17096707 in provider astro 2024-03-16 00:04:47 proc_lsst.multi:201 [INFO] cancelling 17096707 on provider astro proc_lsst.multi INFO: cancelling 17096707 on provider astro 2024-03-16 00:04:47 proc_lsst.multi:146 [INFO] found job 17096725 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096725 in provider escience_rtx6k 2024-03-16 00:04:47 proc_lsst.multi:201 [INFO] cancelling 17096725 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096725 on provider escience_rtx6k 2024-03-16 00:04:47 proc_lsst.multi:146 [INFO] found job 17096709 in provider astro proc_lsst.multi INFO: found job 17096709 in provider astro 2024-03-16 00:04:47 proc_lsst.multi:201 [INFO] cancelling 17096709 on provider astro proc_lsst.multi INFO: cancelling 17096709 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-16 00:04:48 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-16 00:04:48 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17096695', '17096696', '17096697', '17096698', '17096699', '17096700', '17096701', '17096702', '17096703', '17096704', '17096705', '17096706', '17096707', '17096708', '17096709', '17096710', '17096711', '17096712', '17096713', '17096714', '17096715', '17096716', '17096717', '17096718', '17096719', '17096720', '17096721', '17096722', '17096723', '17096724', '17096730', '17096731', '17096732', '17096733', '17096734', '17096735', '17096736', '17096737', '17096738', '17096739', '17096725', '17096726', '17096727', '17096728', '17096729', '13819'] proc_lsst.multi INFO: new jobs since last cancel ['17096695', '17096696', '17096697', '17096698', '17096699', '17096700', '17096701', '17096702', '17096703', '17096704', '17096705', '17096706', '17096707', '17096708', '17096709', '17096710', '17096711', '17096712', '17096713', '17096714', '17096715', '17096716', '17096717', '17096718', '17096719', '17096720', '17096721', '17096722', '17096723', '17096724', '17096730', '17096731', '17096732', '17096733', '17096734', '17096735', '17096736', '17096737', '17096738', '17096739', '17096725', '17096726', '17096727', '17096728', '17096729', '13819'] 2024-03-16 00:04:48 proc_lsst.multi:146 [INFO] found job 17096695 in provider astro proc_lsst.multi INFO: found job 17096695 in provider astro 2024-03-16 00:04:48 proc_lsst.multi:201 [INFO] cancelling 17096695 on provider astro proc_lsst.multi INFO: cancelling 17096695 on provider astro 2024-03-16 00:04:48 proc_lsst.multi:146 [INFO] found job 17096696 in provider astro proc_lsst.multi INFO: found job 17096696 in provider astro 2024-03-16 00:04:48 proc_lsst.multi:201 [INFO] cancelling 17096696 on provider astro proc_lsst.multi INFO: cancelling 17096696 on provider astro 2024-03-16 00:04:48 proc_lsst.multi:146 [INFO] found job 17096697 in provider astro proc_lsst.multi INFO: found job 17096697 in provider astro 2024-03-16 00:04:48 proc_lsst.multi:201 [INFO] cancelling 17096697 on provider astro proc_lsst.multi INFO: cancelling 17096697 on provider astro 2024-03-16 00:04:48 proc_lsst.multi:146 [INFO] found job 17096698 in provider astro proc_lsst.multi INFO: found job 17096698 in provider astro 2024-03-16 00:04:48 proc_lsst.multi:201 [INFO] cancelling 17096698 on provider astro proc_lsst.multi INFO: cancelling 17096698 on provider astro 2024-03-16 00:04:48 proc_lsst.multi:146 [INFO] found job 17096699 in provider astro proc_lsst.multi INFO: found job 17096699 in provider astro 2024-03-16 00:04:48 proc_lsst.multi:201 [INFO] cancelling 17096699 on provider astro proc_lsst.multi INFO: cancelling 17096699 on provider astro 2024-03-16 00:04:49 proc_lsst.multi:146 [INFO] found job 17096700 in provider astro proc_lsst.multi INFO: found job 17096700 in provider astro 2024-03-16 00:04:49 proc_lsst.multi:201 [INFO] cancelling 17096700 on provider astro proc_lsst.multi INFO: cancelling 17096700 on provider astro 2024-03-16 00:04:49 proc_lsst.multi:146 [INFO] found job 17096701 in provider astro proc_lsst.multi INFO: found job 17096701 in provider astro 2024-03-16 00:04:49 proc_lsst.multi:201 [INFO] cancelling 17096701 on provider astro proc_lsst.multi INFO: cancelling 17096701 on provider astro 2024-03-16 00:04:49 proc_lsst.multi:146 [INFO] found job 17096702 in provider astro proc_lsst.multi INFO: found job 17096702 in provider astro 2024-03-16 00:04:49 proc_lsst.multi:201 [INFO] cancelling 17096702 on provider astro proc_lsst.multi INFO: cancelling 17096702 on provider astro 2024-03-16 00:04:49 proc_lsst.multi:146 [INFO] found job 17096703 in provider astro proc_lsst.multi INFO: found job 17096703 in provider astro 2024-03-16 00:04:49 proc_lsst.multi:201 [INFO] cancelling 17096703 on provider astro proc_lsst.multi INFO: cancelling 17096703 on provider astro 2024-03-16 00:04:50 proc_lsst.multi:146 [INFO] found job 17096704 in provider astro proc_lsst.multi INFO: found job 17096704 in provider astro 2024-03-16 00:04:50 proc_lsst.multi:201 [INFO] cancelling 17096704 on provider astro proc_lsst.multi INFO: cancelling 17096704 on provider astro 2024-03-16 00:04:50 proc_lsst.multi:146 [INFO] found job 17096705 in provider astro proc_lsst.multi INFO: found job 17096705 in provider astro 2024-03-16 00:04:50 proc_lsst.multi:201 [INFO] cancelling 17096705 on provider astro proc_lsst.multi INFO: cancelling 17096705 on provider astro 2024-03-16 00:04:50 proc_lsst.multi:146 [INFO] found job 17096706 in provider astro proc_lsst.multi INFO: found job 17096706 in provider astro 2024-03-16 00:04:50 proc_lsst.multi:201 [INFO] cancelling 17096706 on provider astro proc_lsst.multi INFO: cancelling 17096706 on provider astro 2024-03-16 00:04:50 proc_lsst.multi:146 [INFO] found job 17096707 in provider astro proc_lsst.multi INFO: found job 17096707 in provider astro 2024-03-16 00:04:50 proc_lsst.multi:201 [INFO] cancelling 17096707 on provider astro proc_lsst.multi INFO: cancelling 17096707 on provider astro 2024-03-16 00:04:50 proc_lsst.multi:146 [INFO] found job 17096708 in provider astro proc_lsst.multi INFO: found job 17096708 in provider astro 2024-03-16 00:04:50 proc_lsst.multi:201 [INFO] cancelling 17096708 on provider astro proc_lsst.multi INFO: cancelling 17096708 on provider astro 2024-03-16 00:04:50 proc_lsst.multi:146 [INFO] found job 17096709 in provider astro proc_lsst.multi INFO: found job 17096709 in provider astro 2024-03-16 00:04:50 proc_lsst.multi:201 [INFO] cancelling 17096709 on provider astro proc_lsst.multi INFO: cancelling 17096709 on provider astro 2024-03-16 00:04:51 proc_lsst.multi:146 [INFO] found job 17096710 in provider astro proc_lsst.multi INFO: found job 17096710 in provider astro 2024-03-16 00:04:51 proc_lsst.multi:201 [INFO] cancelling 17096710 on provider astro proc_lsst.multi INFO: cancelling 17096710 on provider astro 2024-03-16 00:04:51 proc_lsst.multi:146 [INFO] found job 17096711 in provider astro proc_lsst.multi INFO: found job 17096711 in provider astro 2024-03-16 00:04:51 proc_lsst.multi:201 [INFO] cancelling 17096711 on provider astro proc_lsst.multi INFO: cancelling 17096711 on provider astro 2024-03-16 00:04:51 proc_lsst.multi:146 [INFO] found job 17096712 in provider astro proc_lsst.multi INFO: found job 17096712 in provider astro 2024-03-16 00:04:51 proc_lsst.multi:201 [INFO] cancelling 17096712 on provider astro proc_lsst.multi INFO: cancelling 17096712 on provider astro 2024-03-16 00:04:51 proc_lsst.multi:146 [INFO] found job 17096713 in provider astro proc_lsst.multi INFO: found job 17096713 in provider astro 2024-03-16 00:04:51 proc_lsst.multi:201 [INFO] cancelling 17096713 on provider astro proc_lsst.multi INFO: cancelling 17096713 on provider astro 2024-03-16 00:04:51 proc_lsst.multi:146 [INFO] found job 17096714 in provider astro proc_lsst.multi INFO: found job 17096714 in provider astro 2024-03-16 00:04:51 proc_lsst.multi:201 [INFO] cancelling 17096714 on provider astro proc_lsst.multi INFO: cancelling 17096714 on provider astro 2024-03-16 00:04:52 proc_lsst.multi:146 [INFO] found job 17096715 in provider astro proc_lsst.multi INFO: found job 17096715 in provider astro 2024-03-16 00:04:52 proc_lsst.multi:201 [INFO] cancelling 17096715 on provider astro proc_lsst.multi INFO: cancelling 17096715 on provider astro 2024-03-16 00:04:52 proc_lsst.multi:146 [INFO] found job 17096716 in provider astro proc_lsst.multi INFO: found job 17096716 in provider astro 2024-03-16 00:04:52 proc_lsst.multi:201 [INFO] cancelling 17096716 on provider astro proc_lsst.multi INFO: cancelling 17096716 on provider astro 2024-03-16 00:04:52 proc_lsst.multi:146 [INFO] found job 17096717 in provider astro proc_lsst.multi INFO: found job 17096717 in provider astro 2024-03-16 00:04:52 proc_lsst.multi:201 [INFO] cancelling 17096717 on provider astro proc_lsst.multi INFO: cancelling 17096717 on provider astro 2024-03-16 00:04:52 proc_lsst.multi:146 [INFO] found job 17096718 in provider astro proc_lsst.multi INFO: found job 17096718 in provider astro 2024-03-16 00:04:52 proc_lsst.multi:201 [INFO] cancelling 17096718 on provider astro proc_lsst.multi INFO: cancelling 17096718 on provider astro 2024-03-16 00:04:52 proc_lsst.multi:146 [INFO] found job 17096719 in provider astro proc_lsst.multi INFO: found job 17096719 in provider astro 2024-03-16 00:04:52 proc_lsst.multi:201 [INFO] cancelling 17096719 on provider astro proc_lsst.multi INFO: cancelling 17096719 on provider astro 2024-03-16 00:04:53 proc_lsst.multi:146 [INFO] found job 17096720 in provider astro proc_lsst.multi INFO: found job 17096720 in provider astro 2024-03-16 00:04:53 proc_lsst.multi:201 [INFO] cancelling 17096720 on provider astro proc_lsst.multi INFO: cancelling 17096720 on provider astro 2024-03-16 00:04:53 proc_lsst.multi:146 [INFO] found job 17096721 in provider astro proc_lsst.multi INFO: found job 17096721 in provider astro 2024-03-16 00:04:53 proc_lsst.multi:201 [INFO] cancelling 17096721 on provider astro proc_lsst.multi INFO: cancelling 17096721 on provider astro 2024-03-16 00:04:53 proc_lsst.multi:146 [INFO] found job 17096722 in provider astro proc_lsst.multi INFO: found job 17096722 in provider astro 2024-03-16 00:04:53 proc_lsst.multi:201 [INFO] cancelling 17096722 on provider astro proc_lsst.multi INFO: cancelling 17096722 on provider astro 2024-03-16 00:04:53 proc_lsst.multi:146 [INFO] found job 17096723 in provider astro proc_lsst.multi INFO: found job 17096723 in provider astro 2024-03-16 00:04:53 proc_lsst.multi:201 [INFO] cancelling 17096723 on provider astro proc_lsst.multi INFO: cancelling 17096723 on provider astro 2024-03-16 00:04:53 proc_lsst.multi:146 [INFO] found job 17096724 in provider astro proc_lsst.multi INFO: found job 17096724 in provider astro 2024-03-16 00:04:53 proc_lsst.multi:201 [INFO] cancelling 17096724 on provider astro proc_lsst.multi INFO: cancelling 17096724 on provider astro 2024-03-16 00:04:53 proc_lsst.multi:146 [INFO] found job 17096730 in provider escience_a40 proc_lsst.multi INFO: found job 17096730 in provider escience_a40 2024-03-16 00:04:53 proc_lsst.multi:201 [INFO] cancelling 17096730 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096730 on provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:146 [INFO] found job 17096731 in provider escience_a40 proc_lsst.multi INFO: found job 17096731 in provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:201 [INFO] cancelling 17096731 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096731 on provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:146 [INFO] found job 17096732 in provider escience_a40 proc_lsst.multi INFO: found job 17096732 in provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:201 [INFO] cancelling 17096732 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096732 on provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:146 [INFO] found job 17096733 in provider escience_a40 proc_lsst.multi INFO: found job 17096733 in provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:201 [INFO] cancelling 17096733 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096733 on provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:146 [INFO] found job 17096734 in provider escience_a40 proc_lsst.multi INFO: found job 17096734 in provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:201 [INFO] cancelling 17096734 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096734 on provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:146 [INFO] found job 17096735 in provider escience_a40 proc_lsst.multi INFO: found job 17096735 in provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:201 [INFO] cancelling 17096735 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096735 on provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:146 [INFO] found job 17096736 in provider escience_a40 proc_lsst.multi INFO: found job 17096736 in provider escience_a40 2024-03-16 00:04:54 proc_lsst.multi:201 [INFO] cancelling 17096736 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096736 on provider escience_a40 2024-03-16 00:04:55 proc_lsst.multi:146 [INFO] found job 17096737 in provider escience_a40 proc_lsst.multi INFO: found job 17096737 in provider escience_a40 2024-03-16 00:04:55 proc_lsst.multi:201 [INFO] cancelling 17096737 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096737 on provider escience_a40 2024-03-16 00:04:55 proc_lsst.multi:146 [INFO] found job 17096738 in provider escience_a40 proc_lsst.multi INFO: found job 17096738 in provider escience_a40 2024-03-16 00:04:55 proc_lsst.multi:201 [INFO] cancelling 17096738 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096738 on provider escience_a40 2024-03-16 00:04:55 proc_lsst.multi:146 [INFO] found job 17096739 in provider escience_a40 proc_lsst.multi INFO: found job 17096739 in provider escience_a40 2024-03-16 00:04:55 proc_lsst.multi:201 [INFO] cancelling 17096739 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096739 on provider escience_a40 2024-03-16 00:04:55 proc_lsst.multi:146 [INFO] found job 17096725 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096725 in provider escience_rtx6k 2024-03-16 00:04:55 proc_lsst.multi:201 [INFO] cancelling 17096725 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096725 on provider escience_rtx6k 2024-03-16 00:04:55 proc_lsst.multi:146 [INFO] found job 17096726 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096726 in provider escience_rtx6k 2024-03-16 00:04:55 proc_lsst.multi:201 [INFO] cancelling 17096726 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096726 on provider escience_rtx6k 2024-03-16 00:04:55 proc_lsst.multi:146 [INFO] found job 17096727 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096727 in provider escience_rtx6k 2024-03-16 00:04:55 proc_lsst.multi:201 [INFO] cancelling 17096727 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096727 on provider escience_rtx6k 2024-03-16 00:04:56 proc_lsst.multi:146 [INFO] found job 17096728 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096728 in provider escience_rtx6k 2024-03-16 00:04:56 proc_lsst.multi:201 [INFO] cancelling 17096728 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096728 on provider escience_rtx6k 2024-03-16 00:04:56 proc_lsst.multi:146 [INFO] found job 17096729 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096729 in provider escience_rtx6k 2024-03-16 00:04:56 proc_lsst.multi:201 [INFO] cancelling 17096729 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096729 on provider escience_rtx6k 2024-03-16 00:04:56 proc_lsst.multi:146 [INFO] found job 13819 in provider local proc_lsst.multi INFO: found job 13819 in provider local 2024-03-16 00:04:56 proc_lsst.multi:201 [INFO] cancelling 13819 on provider local proc_lsst.multi INFO: cancelling 13819 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 13819 parsl.providers.local.local WARNING: Failed to kill PID: 13819 and child processes on local 2024-03-16 00:04:57 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 111.2826 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210518_A1e_science#step0_20240316T000255Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 111.2892 seconds; current memory usage: 0.277 Gibyte, delta: 0.006 Gibyte, peak delta: 0.008 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 121.5734 seconds; current memory usage: 0.277 Gibyte, delta: 0.141 Gibyte, peak delta: 0.142 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.279 Gibyte (main), 0.279 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210518_A1e_science#step0_20240316T000255Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m3.464s user 0m30.256s sys 0m13.902s