lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.0975 seconds; current memory usage: 0.177 Gibyte, delta: 0.040 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0106 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0116 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/pipeline_orig.qgraph DEEP/20220822/B1a/science#step0/20240307T041022Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.4710 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 6.4278 seconds; current memory usage: 0.269 Gibyte, delta: 0.092 Gibyte, peak delta: 0.078 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.269 Gibyte (main), 0.261 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0085 seconds; current memory usage: 0.269 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.269 Gibyte (main), 0.261 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 124 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220822_B1a_science#step0_20240307T041022Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0377 seconds; current memory usage: 0.269 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.269 Gibyte (main), 0.261 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 126 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.1892 seconds; current memory usage: 0.270 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.271 Gibyte (main), 0.261 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 5cbbbb7d-de29-4854-a083-2e143a943815 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-07 04:10:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-07 04:10:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 04:10:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-07 04:10:44 proc_lsst.multi:170 [INFO] [multi] job_id 59085 proc_lsst.multi INFO: [multi] job_id 59085 2024-03-07 04:10:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:44 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 59085 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 59085 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 59085 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 4038c995190bf6b579fa66237b1c7ff2 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd96bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b8967ef1-ab63-40df-84b1-a2fb382728d7_isrForCrosstalkSources_1124444_60.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b8967ef1-ab63-40df-84b1-a2fb382728d7_isrForCrosstalkSources_1124444_60.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 56c18f41449d93da7b3f914b642b916f parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/97d33710-128d-4baf-8b72-26e7ac582659_isrForCrosstalkSources_1124444_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/97d33710-128d-4baf-8b72-26e7ac582659_isrForCrosstalkSources_1124444_32.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 7209e7f3680520ba5c425ec34135a78b parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3c1d02a1-a855-4ff3-aa8e-cfbff137065d_isrForCrosstalkSources_1124442_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3c1d02a1-a855-4ff3-aa8e-cfbff137065d_isrForCrosstalkSources_1124442_51.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 9d1143238d8413c3b3882da7b87c4295 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/fd9a8f3e-5824-4832-b0b7-88d1cb210e38_isrForCrosstalkSources_1124444_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/fd9a8f3e-5824-4832-b0b7-88d1cb210e38_isrForCrosstalkSources_1124444_17.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 85f40cef902bcf6a27e7aa7553cf9694 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9e64bb39-0b7c-4081-8f13-2595cf97cfdb_isrForCrosstalkSources_1124444_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9e64bb39-0b7c-4081-8f13-2595cf97cfdb_isrForCrosstalkSources_1124444_27.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 523c6e42321171afa5bf6fe1a91e5380 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/e02e5020-d19f-466f-9868-56370873c664_isrForCrosstalkSources_1124442_34.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/e02e5020-d19f-466f-9868-56370873c664_isrForCrosstalkSources_1124442_34.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 2899dcf2310f4830847e1e0d6b64eb44 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd96b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9566eedd-c57c-4b4e-ac88-44d9b6fd2180_isrForCrosstalkSources_1124444_36.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9566eedd-c57c-4b4e-ac88-44d9b6fd2180_isrForCrosstalkSources_1124444_36.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 21c92c13366e3ceb1718749a22526bef parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/1989d508-d417-43f8-876f-e6baa36eb653_isrForCrosstalkSources_1124442_16.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/1989d508-d417-43f8-876f-e6baa36eb653_isrForCrosstalkSources_1124442_16.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 57e60162693320f892fddd5ddc68b329 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/61a3aa32-c447-4381-8a17-d2338325f5fc_isrForCrosstalkSources_1124442_7.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/61a3aa32-c447-4381-8a17-d2338325f5fc_isrForCrosstalkSources_1124442_7.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 5bc3618d672cc13bca5f9fd76b78958e parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0624f8f3-db97-4f78-8ce9-58b7a474705c_isrForCrosstalkSources_1124444_56.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0624f8f3-db97-4f78-8ce9-58b7a474705c_isrForCrosstalkSources_1124444_56.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash f73cef15f90035ab035f85ff233d2e3a parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/80458c4d-e813-4703-876e-67fe4964c7ee_isrForCrosstalkSources_1124444_18.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/80458c4d-e813-4703-876e-67fe4964c7ee_isrForCrosstalkSources_1124444_18.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash f6fec459b9d422260b0b88fb023465a9 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/63f647aa-82e2-4e5b-84ba-739f3d4b4736_isrForCrosstalkSources_1124442_38.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/63f647aa-82e2-4e5b-84ba-739f3d4b4736_isrForCrosstalkSources_1124442_38.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash b03f906d60b963464ce569d15c632fe7 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e2d0722e-5837-4bcd-8ac3-b7edd109818d_isrForCrosstalkSources_1124444_3.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e2d0722e-5837-4bcd-8ac3-b7edd109818d_isrForCrosstalkSources_1124444_3.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash f7f973df3b13918bf5ffaf29de0706ab parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/fb4613c3-fc04-4d7b-86dd-c155b398801b_isrForCrosstalkSources_1124442_5.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/fb4613c3-fc04-4d7b-86dd-c155b398801b_isrForCrosstalkSources_1124442_5.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash d851e52667b15ebd61916da991381cdc parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/8bd0070d-cead-49f1-b944-455fe575d88e_isrForCrosstalkSources_1124442_33.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/8bd0070d-cead-49f1-b944-455fe575d88e_isrForCrosstalkSources_1124442_33.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash f7fa9dc54f5e39a806f221c452b1147f parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/aa76a740-2eef-46fd-a567-2e42d3cf162b_isrForCrosstalkSources_1124444_19.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/aa76a740-2eef-46fd-a567-2e42d3cf162b_isrForCrosstalkSources_1124444_19.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 0cc3280c3c6bf5c53897798eeb791a43 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/608cdfb7-9368-4fd4-b7c0-3477c8c2d790_isrForCrosstalkSources_1124444_15.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/608cdfb7-9368-4fd4-b7c0-3477c8c2d790_isrForCrosstalkSources_1124444_15.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 139a6e5a10c9301ce62e28d8e6911502 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d291b958-f9f5-46d2-bb02-532d3533f78d_isrForCrosstalkSources_1124442_45.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d291b958-f9f5-46d2-bb02-532d3533f78d_isrForCrosstalkSources_1124442_45.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 6bf9b8c7616ad001223f81b5d7a5e61f parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/a38fc88a-4421-4899-a743-ff636c9a0761_isrForCrosstalkSources_1124444_7.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/a38fc88a-4421-4899-a743-ff636c9a0761_isrForCrosstalkSources_1124444_7.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash ace7cb8f194289f328de82e93207eaf1 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d087f34f-bcf0-4b87-9ea3-b1596a7369e9_isrForCrosstalkSources_1124442_19.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d087f34f-bcf0-4b87-9ea3-b1596a7369e9_isrForCrosstalkSources_1124442_19.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 851ec338672b1fbf56543e9f94f183ad parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/217f7626-edb2-4b48-b9b3-4711b97cf40d_isrForCrosstalkSources_1124444_21.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/217f7626-edb2-4b48-b9b3-4711b97cf40d_isrForCrosstalkSources_1124444_21.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 500bc466f42879f7eeccfaacb26bff41 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d2940c25-e782-457d-9956-9999eb8c6e39_isrForCrosstalkSources_1124442_3.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d2940c25-e782-457d-9956-9999eb8c6e39_isrForCrosstalkSources_1124442_3.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash a5efa4136e442f355f8aca5e8b2a8d37 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/05ce85ce-7372-4c04-8274-73ff29874238_isrForCrosstalkSources_1124442_59.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/05ce85ce-7372-4c04-8274-73ff29874238_isrForCrosstalkSources_1124442_59.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash e681589af10a8eedab2ec5c1f90e5908 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c3382ede-885e-44c6-85e6-d70bf05ba1fb_isrForCrosstalkSources_1124444_41.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c3382ede-885e-44c6-85e6-d70bf05ba1fb_isrForCrosstalkSources_1124444_41.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash bea0ee7ded39dd9f91f20dde284e3475 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/7d72c345-627e-42ae-a883-fd5ca4d8eacc_isrForCrosstalkSources_1124444_14.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/7d72c345-627e-42ae-a883-fd5ca4d8eacc_isrForCrosstalkSources_1124444_14.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 291904314154669a02a0941a5d9f1626 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/47347ce5-e3ab-4934-bd71-6b283c754ac2_isrForCrosstalkSources_1124442_24.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/47347ce5-e3ab-4934-bd71-6b283c754ac2_isrForCrosstalkSources_1124442_24.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash c416cfcb4c18d8566fdd04940bfae0d2 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5d53911e-138d-47bf-9c29-31acfd87dc66_isrForCrosstalkSources_1124442_53.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5d53911e-138d-47bf-9c29-31acfd87dc66_isrForCrosstalkSources_1124442_53.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 0fb983bc77626c88fa3c4a493da2315c parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/17de3b51-45c1-4b4b-b14c-2a70be1fce1b_isrForCrosstalkSources_1124444_1.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/17de3b51-45c1-4b4b-b14c-2a70be1fce1b_isrForCrosstalkSources_1124444_1.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 3b3ec9b5062fb16d760ee3925e32e6d9 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/091dc206-8788-4056-9e81-92b9efe6c3ce_isrForCrosstalkSources_1124444_8.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/091dc206-8788-4056-9e81-92b9efe6c3ce_isrForCrosstalkSources_1124444_8.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash df5a2c01bcf1090a81d1f86aa7f4f1d2 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/9686d9b1-4892-48f3-bd82-c38e1170d8c3_isrForCrosstalkSources_1124442_60.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/9686d9b1-4892-48f3-bd82-c38e1170d8c3_isrForCrosstalkSources_1124442_60.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash aef30bceb3d9e18ca4264b9b12cea197 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/110a6e77-f243-48b1-b938-698da46f98f3_isrForCrosstalkSources_1124444_43.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/110a6e77-f243-48b1-b938-698da46f98f3_isrForCrosstalkSources_1124444_43.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash ee77b677850c014bba88e144616f0e13 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3c01704b-1a47-465e-ba1a-b1fdaef86984_isrForCrosstalkSources_1124442_30.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3c01704b-1a47-465e-ba1a-b1fdaef86984_isrForCrosstalkSources_1124442_30.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 45e0ee936ed3ad4a73b60599099bbf78 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/36248fc5-8fa5-47cf-b9fa-83a268026ffe_isrForCrosstalkSources_1124442_32.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/36248fc5-8fa5-47cf-b9fa-83a268026ffe_isrForCrosstalkSources_1124442_32.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash cc51e1e39eead9d2802f15b4467dc745 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b0743336-27fe-46c6-b71a-746f9c4cb429_isrForCrosstalkSources_1124444_59.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b0743336-27fe-46c6-b71a-746f9c4cb429_isrForCrosstalkSources_1124444_59.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 4a8f84063b14d4257d78e48860ca0f9f parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9bd4f3ac-533a-4c41-8b8b-88c7302dc05e_isrForCrosstalkSources_1124444_38.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9bd4f3ac-533a-4c41-8b8b-88c7302dc05e_isrForCrosstalkSources_1124444_38.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 33797cf832207cac186e024fe16c671f parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d5297e61-6cf6-4836-9646-cdc9025058aa_isrForCrosstalkSources_1124444_49.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d5297e61-6cf6-4836-9646-cdc9025058aa_isrForCrosstalkSources_1124444_49.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 3452c4c210674a14525c3e4b2f9a336c parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ba0585c9-840f-4a10-9a08-a66b788fdbe6_isrForCrosstalkSources_1124442_26.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ba0585c9-840f-4a10-9a08-a66b788fdbe6_isrForCrosstalkSources_1124442_26.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash fde73999232734139ab229c4c4692ce1 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/cb7726ac-1896-4583-9b5c-4ac0a401544a_isrForCrosstalkSources_1124444_26.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/cb7726ac-1896-4583-9b5c-4ac0a401544a_isrForCrosstalkSources_1124444_26.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash c766664765259d0605a05ed522fe82e2 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/65f6e188-34bd-4409-8564-f279c1b87236_isrForCrosstalkSources_1124444_9.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/65f6e188-34bd-4409-8564-f279c1b87236_isrForCrosstalkSources_1124444_9.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash eec8e7e9fd43de25fda70589ae34fec9 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0453c7aa-f480-44fa-8458-d3fc0e6fb4d1_isrForCrosstalkSources_1124444_30.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0453c7aa-f480-44fa-8458-d3fc0e6fb4d1_isrForCrosstalkSources_1124444_30.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 1e7d62590dda0dc8f2c33e4ecd941c2c parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/407980b2-e881-4916-a991-7b6cdfb04ded_isrForCrosstalkSources_1124442_22.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/407980b2-e881-4916-a991-7b6cdfb04ded_isrForCrosstalkSources_1124442_22.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 9c8b623f1b46ffb2607598983cd29c8f parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b3668a60-ad2b-44ff-a9fe-ca33150eae8c_isrForCrosstalkSources_1124444_13.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b3668a60-ad2b-44ff-a9fe-ca33150eae8c_isrForCrosstalkSources_1124444_13.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 3961c1e2068d739b04d1552e446f046d parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/3439ab4a-a3d9-4ff2-a485-5250906ee718_isrForCrosstalkSources_1124444_28.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/3439ab4a-a3d9-4ff2-a485-5250906ee718_isrForCrosstalkSources_1124444_28.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 3fd804843eef17d750cd28649996e95d parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/f170cbcf-2a73-4e8d-8f09-56168f358616_isrForCrosstalkSources_1124442_15.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/f170cbcf-2a73-4e8d-8f09-56168f358616_isrForCrosstalkSources_1124442_15.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash d2eec9dfbfa457efd296e4b27c01879e parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/6529552a-7251-4ade-b78f-c8c3831de7af_isrForCrosstalkSources_1124444_62.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/6529552a-7251-4ade-b78f-c8c3831de7af_isrForCrosstalkSources_1124444_62.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 87c80219f267bc178735581074eb7337 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/033029c5-80ce-4775-8b7b-926d45827b13_isrForCrosstalkSources_1124442_6.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/033029c5-80ce-4775-8b7b-926d45827b13_isrForCrosstalkSources_1124442_6.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash a3a8bf7dc61f2af36d385545fab8dc54 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/8503dcd0-c5e0-432d-a9be-b7a51728a337_isrForCrosstalkSources_1124442_52.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/8503dcd0-c5e0-432d-a9be-b7a51728a337_isrForCrosstalkSources_1124442_52.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 3661abeaa55e71ec38e6b7d0a256f0bc parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c1187ceb-2f79-4ec3-af7c-634cbd61b481_isrForCrosstalkSources_1124444_40.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c1187ceb-2f79-4ec3-af7c-634cbd61b481_isrForCrosstalkSources_1124444_40.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 918494800ec8ffb42efeb01e0e95b393 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d034c157-fd4e-4b03-94c7-224ffb51107f_isrForCrosstalkSources_1124444_51.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d034c157-fd4e-4b03-94c7-224ffb51107f_isrForCrosstalkSources_1124444_51.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash edb89f612eb94d7870a9ba0461aef951 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/10c111b8-36ef-4714-9759-81449226ae37_isrForCrosstalkSources_1124444_24.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/10c111b8-36ef-4714-9759-81449226ae37_isrForCrosstalkSources_1124444_24.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash eb135d87ef9a6fccc16caa68adac24c6 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/64a3f1e4-f3ce-4903-9371-8ca3d26576ac_isrForCrosstalkSources_1124442_28.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/64a3f1e4-f3ce-4903-9371-8ca3d26576ac_isrForCrosstalkSources_1124442_28.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 5c011be69ec6e42b0be6520b5de82037 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3813a944-a7f4-4fbb-b0bb-0b821388def7_isrForCrosstalkSources_1124442_48.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3813a944-a7f4-4fbb-b0bb-0b821388def7_isrForCrosstalkSources_1124442_48.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 73618202eec041b9a521fe155c113da7 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/a0fe2fd9-3564-40a3-9777-1b91c657e3a2_isrForCrosstalkSources_1124442_57.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/a0fe2fd9-3564-40a3-9777-1b91c657e3a2_isrForCrosstalkSources_1124442_57.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 876e3e10f1439f8aa6ebe7c4a5adb43d parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/595d2f09-7c62-4729-93f2-e19f1fa84f84_isrForCrosstalkSources_1124444_39.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/595d2f09-7c62-4729-93f2-e19f1fa84f84_isrForCrosstalkSources_1124444_39.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash ce8f0bb389e3192b25890bc7188cb75b parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6bf85c17-0032-44e9-a786-212d01195984_isrForCrosstalkSources_1124442_50.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6bf85c17-0032-44e9-a786-212d01195984_isrForCrosstalkSources_1124442_50.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash b4a2fda805c04d4a1a9fb84eafbb3833 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/1347c6ba-f739-4bbd-834c-3ef9d1e6fefd_isrForCrosstalkSources_1124444_58.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/1347c6ba-f739-4bbd-834c-3ef9d1e6fefd_isrForCrosstalkSources_1124444_58.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 9267b6a478736dee7bfb13b7d0b7101e parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/6e16cc32-b8de-48c3-adf7-082ec399aab6_isrForCrosstalkSources_1124444_12.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/6e16cc32-b8de-48c3-adf7-082ec399aab6_isrForCrosstalkSources_1124444_12.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 71b7018050f5bbaddc0726d8117d6bec parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/de816d44-6691-4376-acd3-cf5904f19d09_isrForCrosstalkSources_1124442_18.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/de816d44-6691-4376-acd3-cf5904f19d09_isrForCrosstalkSources_1124442_18.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 65a0a8988020ed4578f4630d631fce71 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5d646156-bf97-4902-9c63-ed837b6140cd_isrForCrosstalkSources_1124442_10.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5d646156-bf97-4902-9c63-ed837b6140cd_isrForCrosstalkSources_1124442_10.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash f16e6cef698724c5cae47b9eaf3fc5ae parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5c207647-56ce-430d-9964-1839e87e5e7b_isrForCrosstalkSources_1124442_9.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5c207647-56ce-430d-9964-1839e87e5e7b_isrForCrosstalkSources_1124442_9.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash ab7bd91a9f49b5d455e6e015e4229119 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/c67278a2-c331-437a-bd7c-258c0e80d4f8_isrForCrosstalkSources_1124442_23.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/c67278a2-c331-437a-bd7c-258c0e80d4f8_isrForCrosstalkSources_1124442_23.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 36f3bfa1e6632f014ae2b059b596878d parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/4283937f-7a04-44be-8262-a7842fa502fd_isrForCrosstalkSources_1124442_39.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/4283937f-7a04-44be-8262-a7842fa502fd_isrForCrosstalkSources_1124442_39.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 31c2ceb1ac051a18fc994939ae0f3010 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/0b26a447-671b-405e-b6cc-e788b0299c0b_isrForCrosstalkSources_1124442_21.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/0b26a447-671b-405e-b6cc-e788b0299c0b_isrForCrosstalkSources_1124442_21.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 94df266d263c4de93152e350a71b30f2 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/0f48fa49-5c76-46dc-b9f2-958a74ec381d_isrForCrosstalkSources_1124442_41.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/0f48fa49-5c76-46dc-b9f2-958a74ec381d_isrForCrosstalkSources_1124442_41.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash e35579c1d54d299eaf2d44d6085f0409 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0b0fa0a5-b414-426b-be5b-9806113f0dae_isrForCrosstalkSources_1124444_20.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0b0fa0a5-b414-426b-be5b-9806113f0dae_isrForCrosstalkSources_1124444_20.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash a8fe019b6fc52789de1df7cb040fccc6 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/029fc1d3-9407-4ebe-8483-038bec0f511b_isrForCrosstalkSources_1124442_43.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/029fc1d3-9407-4ebe-8483-038bec0f511b_isrForCrosstalkSources_1124442_43.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash cf32a9bbc0e7638f815fcd8cb4d6d7ac parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/18e8b1e9-883e-481c-8292-664fe15efe2a_isrForCrosstalkSources_1124444_31.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/18e8b1e9-883e-481c-8292-664fe15efe2a_isrForCrosstalkSources_1124444_31.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 86351204f95fe6787cb6347ed0c6b776 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/e5af3728-598e-438c-8297-fbf5dc29827f_isrForCrosstalkSources_1124442_13.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/e5af3728-598e-438c-8297-fbf5dc29827f_isrForCrosstalkSources_1124442_13.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash fdebde54d68b1e8dcfc4591e8562edda parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0a55c0f4-ebec-4226-ae0e-3cb014573df5_isrForCrosstalkSources_1124444_44.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0a55c0f4-ebec-4226-ae0e-3cb014573df5_isrForCrosstalkSources_1124444_44.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash d28e4dc7ed2da9164e48e45b02a3b9de parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/a2de857e-016f-4e59-8d10-71c340e3d63b_isrForCrosstalkSources_1124442_54.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/a2de857e-016f-4e59-8d10-71c340e3d63b_isrForCrosstalkSources_1124442_54.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 341426370b51fe6a7552116bc9895fe7 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ac87da8c-20a0-472a-8c48-34dafb7e87b7_isrForCrosstalkSources_1124444_23.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ac87da8c-20a0-472a-8c48-34dafb7e87b7_isrForCrosstalkSources_1124444_23.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 479cdd9899f7af0b36385e94f28e8ca1 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3ce045db-0d60-4273-84d6-d5a0513af790_isrForCrosstalkSources_1124442_44.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3ce045db-0d60-4273-84d6-d5a0513af790_isrForCrosstalkSources_1124442_44.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 839ac9937a8ee28f6098f06764f824d1 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/cae58037-6795-45c0-acc9-bddeeb327109_isrForCrosstalkSources_1124444_5.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/cae58037-6795-45c0-acc9-bddeeb327109_isrForCrosstalkSources_1124444_5.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash ee043d6161b32dbe9337ae4e35473aad parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/145d2e8b-a2d3-4cb6-91cc-525f16877f88_isrForCrosstalkSources_1124442_17.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/145d2e8b-a2d3-4cb6-91cc-525f16877f88_isrForCrosstalkSources_1124442_17.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash a4a563a574003e73b5e4fe8c207389a7 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6f967276-b778-4fb6-9590-29cbf816d379_isrForCrosstalkSources_1124442_56.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6f967276-b778-4fb6-9590-29cbf816d379_isrForCrosstalkSources_1124442_56.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash b5def5013aaad238adf1cec95bfd1b3d parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/95071f74-7345-467c-9b3c-658a5b211b45_isrForCrosstalkSources_1124444_34.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/95071f74-7345-467c-9b3c-658a5b211b45_isrForCrosstalkSources_1124444_34.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 46f02d97794be9f5e7cca65b1bb347e4 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/20547dae-7e85-4c33-932c-2348183191ac_isrForCrosstalkSources_1124442_35.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/20547dae-7e85-4c33-932c-2348183191ac_isrForCrosstalkSources_1124442_35.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash d2a54c0671854e858ea2ace51e6a6049 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/bbfc14ff-1c31-4c8d-87ff-d412ddcc18b5_isrForCrosstalkSources_1124442_27.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/bbfc14ff-1c31-4c8d-87ff-d412ddcc18b5_isrForCrosstalkSources_1124442_27.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 483f28f4609527fff4c0b8603aa11420 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/1ad6ac9e-8a08-43c6-9740-26b63218c714_isrForCrosstalkSources_1124442_49.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/1ad6ac9e-8a08-43c6-9740-26b63218c714_isrForCrosstalkSources_1124442_49.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 17dd1d15a64fc49e1eb0cfff30b1fdc9 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/eaa4379d-b15c-4b5c-988f-e8737d7eac36_isrForCrosstalkSources_1124444_2.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/eaa4379d-b15c-4b5c-988f-e8737d7eac36_isrForCrosstalkSources_1124444_2.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 97834d10a7157f5d02b5005dad979592 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/73e42271-1adc-4d34-a1d9-8cf8f31e4f95_isrForCrosstalkSources_1124444_50.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/73e42271-1adc-4d34-a1d9-8cf8f31e4f95_isrForCrosstalkSources_1124444_50.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 7360c15e42fc09d8709a2072e164f8b5 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/697ac075-8814-43f3-abc4-2a66df22e1a6_isrForCrosstalkSources_1124442_20.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/697ac075-8814-43f3-abc4-2a66df22e1a6_isrForCrosstalkSources_1124442_20.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash f50c37036ed2d70c94bc40ab76843b21 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/589994e1-455c-4c34-8bb6-45bff732c5a5_isrForCrosstalkSources_1124442_8.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/589994e1-455c-4c34-8bb6-45bff732c5a5_isrForCrosstalkSources_1124442_8.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 075d7ed12b2fd14b5b07b266801818d1 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/5763aea6-8d3a-4619-8355-27efc6a2d07e_isrForCrosstalkSources_1124444_48.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/5763aea6-8d3a-4619-8355-27efc6a2d07e_isrForCrosstalkSources_1124444_48.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 96f8f81032256e12a7303794715dbe81 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/42de30c1-d7a4-4f71-a27e-5b7ed97163c3_isrForCrosstalkSources_1124442_31.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/42de30c1-d7a4-4f71-a27e-5b7ed97163c3_isrForCrosstalkSources_1124442_31.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 48a0db2c42a7bc940094c777d98beb8b parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/b9931ca4-dcd7-402e-98db-88bc9da7b3bc_isrForCrosstalkSources_1124442_58.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/b9931ca4-dcd7-402e-98db-88bc9da7b3bc_isrForCrosstalkSources_1124442_58.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 966c8dbe4bd62d5240be7a842bd00f1e parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ed07a291-b2a6-4029-90f0-028f737ad5dc_isrForCrosstalkSources_1124442_37.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ed07a291-b2a6-4029-90f0-028f737ad5dc_isrForCrosstalkSources_1124442_37.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 9be50ba36de379b1b7877cd04acf62ac parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/50051efb-806f-4f18-869d-8ccf9316d891_isrForCrosstalkSources_1124444_45.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/50051efb-806f-4f18-869d-8ccf9316d891_isrForCrosstalkSources_1124444_45.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 7aca312294584ad59852e98f84cb5694 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/2e23dc98-e685-4675-b1fc-c9e4837581b1_isrForCrosstalkSources_1124442_46.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/2e23dc98-e685-4675-b1fc-c9e4837581b1_isrForCrosstalkSources_1124442_46.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 0386c39292040cdf31827b4f4bd9e3cf parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/69b5478d-3cdc-40a3-9baf-acd091560a33_isrForCrosstalkSources_1124442_42.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/69b5478d-3cdc-40a3-9baf-acd091560a33_isrForCrosstalkSources_1124442_42.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 13fb2580a4a14884519bff76846f52cf parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bd9b7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/9e7e54bf-1351-4cb3-9a84-2fe97dca51b2_isrForCrosstalkSources_1124442_1.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/9e7e54bf-1351-4cb3-9a84-2fe97dca51b2_isrForCrosstalkSources_1124442_1.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash a660fc4e09e4b8a2fd3542f88082ef48 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc4040e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0cec5cac-3914-4286-b1c1-0438125e9f4d_isrForCrosstalkSources_1124444_47.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0cec5cac-3914-4286-b1c1-0438125e9f4d_isrForCrosstalkSources_1124444_47.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash fb56373f269ab5bb341232b9c60b7026 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/be8a98d0-aeb2-4709-a5e8-cb687cc24f0d_isrForCrosstalkSources_1124442_61.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/be8a98d0-aeb2-4709-a5e8-cb687cc24f0d_isrForCrosstalkSources_1124442_61.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 1eba89cb3d5aa600b0d5bf47cc8f27e1 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/494fb10d-23f5-4539-a255-881a002bb7ab_isrForCrosstalkSources_1124444_16.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/494fb10d-23f5-4539-a255-881a002bb7ab_isrForCrosstalkSources_1124444_16.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash f50f398cfea855eeabb69fb48bef8ef2 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/a95038da-bbe8-473c-beb0-50f2bd069384_isrForCrosstalkSources_1124444_53.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/a95038da-bbe8-473c-beb0-50f2bd069384_isrForCrosstalkSources_1124444_53.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash b8334f05648cfffe411a49784bba9946 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc4042c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9e95e977-5049-4232-af3e-07bfa8cc25d8_isrForCrosstalkSources_1124444_10.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9e95e977-5049-4232-af3e-07bfa8cc25d8_isrForCrosstalkSources_1124444_10.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash cd74bd8e3ccad951709d3e340eef8acc parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/7486d1b5-5c1e-4381-b7c7-45f58044e36b_isrForCrosstalkSources_1124442_40.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/7486d1b5-5c1e-4381-b7c7-45f58044e36b_isrForCrosstalkSources_1124442_40.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 43fe3f0666b7d7d1599675eb7b89bdd2 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc4044a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/93770227-ae55-403d-bb78-2cef5ee49b86_isrForCrosstalkSources_1124444_25.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/93770227-ae55-403d-bb78-2cef5ee49b86_isrForCrosstalkSources_1124444_25.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 006abf90395beb78203afbdc626499b2 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/7eb3e0ec-1230-4069-af56-be8057d10437_isrForCrosstalkSources_1124444_61.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/7eb3e0ec-1230-4069-af56-be8057d10437_isrForCrosstalkSources_1124444_61.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 4033ebaaa7ea4bcee31af354b8801854 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d8c645f2-5772-4392-b031-d9244f75c4df_isrForCrosstalkSources_1124442_62.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d8c645f2-5772-4392-b031-d9244f75c4df_isrForCrosstalkSources_1124442_62.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 9f645c57df7d691cbb1772c9f386f49f parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc4045e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c6ef7975-6e77-4a55-b74d-eaf5da584ccc_isrForCrosstalkSources_1124444_57.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c6ef7975-6e77-4a55-b74d-eaf5da584ccc_isrForCrosstalkSources_1124444_57.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 21ab8b2e97bef11fb1ec46794c78201b parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ddafa3b0-8369-4af8-b6e1-f9bc9e36d9b5_isrForCrosstalkSources_1124444_11.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ddafa3b0-8369-4af8-b6e1-f9bc9e36d9b5_isrForCrosstalkSources_1124444_11.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 7cff50c45c3ab2445d38d70f9b2c9353 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc4047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d99c318c-fc92-4676-af6f-2134918ef1ca_isrForCrosstalkSources_1124442_47.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d99c318c-fc92-4676-af6f-2134918ef1ca_isrForCrosstalkSources_1124442_47.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash c2c549166ac9188f5b5a0a290dd2f520 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/8c0ec435-1811-4ffb-8a8e-c8b7f94283bb_isrForCrosstalkSources_1124444_52.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/8c0ec435-1811-4ffb-8a8e-c8b7f94283bb_isrForCrosstalkSources_1124444_52.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 473fa49c094333442ae9edd54af168f3 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc4049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3fcd92bc-3593-4a5c-a9f2-4e2f8a2db0cb_isrForCrosstalkSources_1124442_11.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3fcd92bc-3593-4a5c-a9f2-4e2f8a2db0cb_isrForCrosstalkSources_1124442_11.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 12af00c17c3b84c162fa698bb16fcd8e parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/63727be1-3c1b-49fa-b917-98f5c8bd5bdb_isrForCrosstalkSources_1124444_29.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/63727be1-3c1b-49fa-b917-98f5c8bd5bdb_isrForCrosstalkSources_1124444_29.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 4ef09c48f93491add370d4608f865a03 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e1d117ae-37aa-4dc4-b176-aa3aaa21c170_isrForCrosstalkSources_1124444_37.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e1d117ae-37aa-4dc4-b176-aa3aaa21c170_isrForCrosstalkSources_1124444_37.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 8e9549fcbd2df8546369b9b752b565ea parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0d2c722e-ffd5-498b-aff6-2daad731436e_isrForCrosstalkSources_1124444_54.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0d2c722e-ffd5-498b-aff6-2daad731436e_isrForCrosstalkSources_1124444_54.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash ce8f53fcda7c570de4cac96318cf389b parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/4844f0af-6d63-4286-9ce9-af824883d189_isrForCrosstalkSources_1124442_36.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/4844f0af-6d63-4286-9ce9-af824883d189_isrForCrosstalkSources_1124442_36.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash e4110d691aff7a265dc09996da681f53 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/61aa40c0-c887-4d2f-a0c4-c0483290f63c_isrForCrosstalkSources_1124444_46.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/61aa40c0-c887-4d2f-a0c4-c0483290f63c_isrForCrosstalkSources_1124444_46.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash d8ade0eb212eb53378af9a326bb09c0e parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e9980e8e-681c-4cfa-9f74-7f8238f587c3_isrForCrosstalkSources_1124444_42.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e9980e8e-681c-4cfa-9f74-7f8238f587c3_isrForCrosstalkSources_1124444_42.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 666166337c8f3b6e95e8bfa02ddd6f4c parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e5e0bc4a-f7ef-43e2-a9e9-6e8a2e9dc885_isrForCrosstalkSources_1124444_4.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e5e0bc4a-f7ef-43e2-a9e9-6e8a2e9dc885_isrForCrosstalkSources_1124444_4.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 16bc9d8105340cfee6be0bf725b595f2 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/59c767ff-9529-4673-8e2f-20102917a420_isrForCrosstalkSources_1124444_35.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/59c767ff-9529-4673-8e2f-20102917a420_isrForCrosstalkSources_1124444_35.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash c5aa4a7658b2e53ac4ed03abb7b33800 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/68196536-94cc-41a4-8aa5-974d4052b298_isrForCrosstalkSources_1124442_4.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/68196536-94cc-41a4-8aa5-974d4052b298_isrForCrosstalkSources_1124442_4.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 69bbcede7fe67d1c1702007185fe2813 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/8ceb7a56-193f-4b98-8dd6-546f906b36ef_isrForCrosstalkSources_1124444_55.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/8ceb7a56-193f-4b98-8dd6-546f906b36ef_isrForCrosstalkSources_1124444_55.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash a194a5185f9f9f6c9dd0dd0441f7bf03 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc404f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5dadfcab-7174-4df7-8979-2c0a1e7ac008_isrForCrosstalkSources_1124442_12.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5dadfcab-7174-4df7-8979-2c0a1e7ac008_isrForCrosstalkSources_1124442_12.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 4895dbde1b5221ac245d3af0b777a5bb parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc405080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/2a4e6f2f-055f-4f05-a175-7d2a7d6171a5_isrForCrosstalkSources_1124442_25.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/2a4e6f2f-055f-4f05-a175-7d2a7d6171a5_isrForCrosstalkSources_1124442_25.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 7e5ba4112d34f6da5180a43d5f042228 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc405120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/980b148e-4bb0-417d-97fa-01d11bc387ce_isrForCrosstalkSources_1124442_29.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/980b148e-4bb0-417d-97fa-01d11bc387ce_isrForCrosstalkSources_1124442_29.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash bf612ed3e29491cd2d5a2c1b661322ac parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc4051c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ae84afd4-a62e-4dfe-abc2-808ab7746bc4_isrForCrosstalkSources_1124444_6.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ae84afd4-a62e-4dfe-abc2-808ab7746bc4_isrForCrosstalkSources_1124444_6.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 0210d00f7860fd93566ec473c5e93f5d parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc405300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/191f2053-0ed9-4ff2-a8aa-3496ce93cc49_isrForCrosstalkSources_1124444_22.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/191f2053-0ed9-4ff2-a8aa-3496ce93cc49_isrForCrosstalkSources_1124444_22.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 258f11ed8f447828609182c0d24678f6 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc405260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6497f69f-d326-463a-912c-d896d00a27f7_isrForCrosstalkSources_1124442_14.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6497f69f-d326-463a-912c-d896d00a27f7_isrForCrosstalkSources_1124442_14.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash f3d83e85f222e7ee3ef85f583862ed26 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc4053a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ec04b458-3f04-4b39-aeb3-1567251461df_isrForCrosstalkSources_1124442_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ec04b458-3f04-4b39-aeb3-1567251461df_isrForCrosstalkSources_1124442_2.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 9693dd9e1852a8e1657ddeb75dd5ff03 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc405440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d3a01e21-dd42-46da-91f9-21f6466692e9_isrForCrosstalkSources_1124444_33.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d3a01e21-dd42-46da-91f9-21f6466692e9_isrForCrosstalkSources_1124444_33.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash dbca39250aefc9571f5fefd2f1890cda parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14f0bc4054e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/cd1b5169-d91a-4069-b3fa-5938bea4087b_isrForCrosstalkSources_1124442_55.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/cd1b5169-d91a-4069-b3fa-5938bea4087b_isrForCrosstalkSources_1124442_55.stderr 2024-03-07 04:10:45 proc_lsst.multi:146 [INFO] found job 59085 in provider local proc_lsst.multi INFO: found job 59085 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 124 parsl.dataflow.strategy DEBUG: Executor multi has 124 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 30 more blocks parsl.executors.status_handling INFO: Scaling out by 30 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-07 04:10:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-07 04:10:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 04:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:45 proc_lsst.multi:170 [INFO] [multi] job_id 16884954 proc_lsst.multi INFO: [multi] job_id 16884954 2024-03-07 04:10:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884954 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884954 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16884954 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-07 04:10:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-07 04:10:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:45 proc_lsst.multi:170 [INFO] [multi] job_id 16884955 proc_lsst.multi INFO: [multi] job_id 16884955 2024-03-07 04:10:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 04:10:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884955 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884955 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16884955 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-07 04:10:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-07 04:10:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 04:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:45 proc_lsst.multi:170 [INFO] [multi] job_id 16884956 proc_lsst.multi INFO: [multi] job_id 16884956 2024-03-07 04:10:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 04:10:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884956 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884956 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16884956 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-07 04:10:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-07 04:10:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 04:10:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:46 proc_lsst.multi:170 [INFO] [multi] job_id 16884957 proc_lsst.multi INFO: [multi] job_id 16884957 2024-03-07 04:10:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 04:10:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884957 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884957 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16884957 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-07 04:10:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-07 04:10:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 04:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:46 proc_lsst.multi:170 [INFO] [multi] job_id 16884958 proc_lsst.multi INFO: [multi] job_id 16884958 2024-03-07 04:10:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 04:10:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884958 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884958 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16884958 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-07 04:10:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-07 04:10:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 04:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:46 proc_lsst.multi:170 [INFO] [multi] job_id 16884959 proc_lsst.multi INFO: [multi] job_id 16884959 2024-03-07 04:10:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 04:10:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884959 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884959 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16884959 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-07 04:10:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-07 04:10:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 04:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:46 proc_lsst.multi:170 [INFO] [multi] job_id 16884960 proc_lsst.multi INFO: [multi] job_id 16884960 2024-03-07 04:10:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 04:10:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884960 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884960 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16884960 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-07 04:10:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-07 04:10:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 04:10:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:47 proc_lsst.multi:170 [INFO] [multi] job_id 16884961 proc_lsst.multi INFO: [multi] job_id 16884961 2024-03-07 04:10:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 04:10:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884961 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884961 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16884961 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-07 04:10:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-07 04:10:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 04:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:47 proc_lsst.multi:170 [INFO] [multi] job_id 16884962 proc_lsst.multi INFO: [multi] job_id 16884962 2024-03-07 04:10:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 04:10:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884962 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884962 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16884962 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-07 04:10:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-07 04:10:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 04:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:47 proc_lsst.multi:170 [INFO] [multi] job_id 16884963 proc_lsst.multi INFO: [multi] job_id 16884963 2024-03-07 04:10:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 04:10:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884963 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884963 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16884963 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-07 04:10:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-07 04:10:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 04:10:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:48 proc_lsst.multi:170 [INFO] [multi] job_id 16884964 proc_lsst.multi INFO: [multi] job_id 16884964 2024-03-07 04:10:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 04:10:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884964 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884964 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16884964 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-07 04:10:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-07 04:10:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 04:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:48 proc_lsst.multi:170 [INFO] [multi] job_id 16884965 proc_lsst.multi INFO: [multi] job_id 16884965 2024-03-07 04:10:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 04:10:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884965 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884965 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16884965 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-07 04:10:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-07 04:10:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 04:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:48 proc_lsst.multi:170 [INFO] [multi] job_id 16884966 proc_lsst.multi INFO: [multi] job_id 16884966 2024-03-07 04:10:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 04:10:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884966 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884966 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16884966 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-07 04:10:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-07 04:10:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 04:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:48 proc_lsst.multi:170 [INFO] [multi] job_id 16884967 proc_lsst.multi INFO: [multi] job_id 16884967 2024-03-07 04:10:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 04:10:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884967 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884967 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16884967 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-07 04:10:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-07 04:10:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 04:10:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:49 proc_lsst.multi:170 [INFO] [multi] job_id 16884968 proc_lsst.multi INFO: [multi] job_id 16884968 2024-03-07 04:10:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 04:10:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884968 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884968 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16884968 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-07 04:10:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-07 04:10:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 04:10:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:51 proc_lsst.multi:170 [INFO] [multi] job_id 16884972 proc_lsst.multi INFO: [multi] job_id 16884972 2024-03-07 04:10:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 04:10:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884972 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884972 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16884972 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-07 04:10:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:51 proc_lsst.multi:170 [INFO] [multi] job_id 16884973 proc_lsst.multi INFO: [multi] job_id 16884973 2024-03-07 04:10:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 04:10:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884973 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884973 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16884973 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-07 04:10:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:51 proc_lsst.multi:170 [INFO] [multi] job_id 16884974 proc_lsst.multi INFO: [multi] job_id 16884974 2024-03-07 04:10:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 04:10:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884974 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884974 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16884974 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-07 04:10:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:51 proc_lsst.multi:170 [INFO] [multi] job_id 16884975 proc_lsst.multi INFO: [multi] job_id 16884975 2024-03-07 04:10:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 04:10:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884975 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884975 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16884975 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-07 04:10:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:51 proc_lsst.multi:170 [INFO] [multi] job_id 16884976 proc_lsst.multi INFO: [multi] job_id 16884976 2024-03-07 04:10:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 04:10:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884976 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884976 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16884976 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-07 04:10:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:51 proc_lsst.multi:170 [INFO] [multi] job_id 16884977 proc_lsst.multi INFO: [multi] job_id 16884977 2024-03-07 04:10:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 04:10:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884977 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884977 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16884977 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-07 04:10:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 04:10:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:52 proc_lsst.multi:170 [INFO] [multi] job_id 16884978 proc_lsst.multi INFO: [multi] job_id 16884978 2024-03-07 04:10:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 04:10:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884978 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884978 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16884978 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-07 04:10:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-07 04:10:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 04:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:52 proc_lsst.multi:170 [INFO] [multi] job_id 16884979 proc_lsst.multi INFO: [multi] job_id 16884979 2024-03-07 04:10:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 04:10:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884979 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884979 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16884979 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-07 04:10:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-07 04:10:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 04:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:52 proc_lsst.multi:170 [INFO] [multi] job_id 16884980 proc_lsst.multi INFO: [multi] job_id 16884980 2024-03-07 04:10:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 04:10:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884980 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884980 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16884980 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-07 04:10:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-07 04:10:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 04:10:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:53 proc_lsst.multi:170 [INFO] [multi] job_id 16884981 proc_lsst.multi INFO: [multi] job_id 16884981 2024-03-07 04:10:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 04:10:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884981 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884981 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16884981 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-07 04:10:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-07 04:10:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 04:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:53 proc_lsst.multi:170 [INFO] [multi] job_id 16884982 proc_lsst.multi INFO: [multi] job_id 16884982 2024-03-07 04:10:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 04:10:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884982 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884982 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16884982 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-07 04:10:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-07 04:10:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 04:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:53 proc_lsst.multi:170 [INFO] [multi] job_id 16884983 proc_lsst.multi INFO: [multi] job_id 16884983 2024-03-07 04:10:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 04:10:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884983 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884983 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16884983 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-07 04:10:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-07 04:10:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 04:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:53 proc_lsst.multi:170 [INFO] [multi] job_id 16884984 proc_lsst.multi INFO: [multi] job_id 16884984 2024-03-07 04:10:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 04:10:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884984 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884984 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16884984 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-07 04:10:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-07 04:10:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 04:10:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:54 proc_lsst.multi:170 [INFO] [multi] job_id 16884985 proc_lsst.multi INFO: [multi] job_id 16884985 2024-03-07 04:10:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 04:10:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884985 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884985 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16884985 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-07 04:10:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-07 04:10:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 04:10:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 04:10:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 04:10:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 04:10:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 04:10:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54393 --result_port=54112 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 04:10:54 proc_lsst.multi:170 [INFO] [multi] job_id 16884986 proc_lsst.multi INFO: [multi] job_id 16884986 2024-03-07 04:10:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 04:10:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 04:10:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884986 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884986 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16884986 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 124 parsl.dataflow.strategy DEBUG: Executor multi has 124 active tasks, 1/30 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56c18f41449d93da7b3f914b642b916f with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/97d33710-128d-4baf-8b72-26e7ac582659_isrForCrosstalkSources_1124444_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/97d33710-128d-4baf-8b72-26e7ac582659_isrForCrosstalkSources_1124444_32.stderr parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4038c995190bf6b579fa66237b1c7ff2 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b8967ef1-ab63-40df-84b1-a2fb382728d7_isrForCrosstalkSources_1124444_60.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b8967ef1-ab63-40df-84b1-a2fb382728d7_isrForCrosstalkSources_1124444_60.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 122 parsl.dataflow.strategy DEBUG: Executor multi has 122 active tasks, 1/30 running/pending blocks, and 28 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7209e7f3680520ba5c425ec34135a78b with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3c1d02a1-a855-4ff3-aa8e-cfbff137065d_isrForCrosstalkSources_1124442_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3c1d02a1-a855-4ff3-aa8e-cfbff137065d_isrForCrosstalkSources_1124442_51.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d1143238d8413c3b3882da7b87c4295 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/fd9a8f3e-5824-4832-b0b7-88d1cb210e38_isrForCrosstalkSources_1124444_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/fd9a8f3e-5824-4832-b0b7-88d1cb210e38_isrForCrosstalkSources_1124444_17.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 120 parsl.dataflow.strategy DEBUG: Executor multi has 120 active tasks, 1/30 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 120 parsl.dataflow.strategy DEBUG: Executor multi has 120 active tasks, 1/30 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a8f84063b14d4257d78e48860ca0f9f with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9bd4f3ac-533a-4c41-8b8b-88c7302dc05e_isrForCrosstalkSources_1124444_38.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9bd4f3ac-533a-4c41-8b8b-88c7302dc05e_isrForCrosstalkSources_1124444_38.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33797cf832207cac186e024fe16c671f with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d5297e61-6cf6-4836-9646-cdc9025058aa_isrForCrosstalkSources_1124444_49.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d5297e61-6cf6-4836-9646-cdc9025058aa_isrForCrosstalkSources_1124444_49.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 118 parsl.dataflow.strategy DEBUG: Executor multi has 118 active tasks, 1/30 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3452c4c210674a14525c3e4b2f9a336c with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ba0585c9-840f-4a10-9a08-a66b788fdbe6_isrForCrosstalkSources_1124442_26.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ba0585c9-840f-4a10-9a08-a66b788fdbe6_isrForCrosstalkSources_1124442_26.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fde73999232734139ab229c4c4692ce1 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/cb7726ac-1896-4583-9b5c-4ac0a401544a_isrForCrosstalkSources_1124444_26.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/cb7726ac-1896-4583-9b5c-4ac0a401544a_isrForCrosstalkSources_1124444_26.stderr 2024-03-07 04:11:19 proc_lsst.multi:146 [INFO] found job 59085 in provider local proc_lsst.multi INFO: found job 59085 in provider local 2024-03-07 04:11:19 proc_lsst.multi:146 [INFO] found job 16884954 in provider astro proc_lsst.multi INFO: found job 16884954 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:19 proc_lsst.multi:146 [INFO] found job 16884955 in provider astro proc_lsst.multi INFO: found job 16884955 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:20 proc_lsst.multi:146 [INFO] found job 16884956 in provider astro proc_lsst.multi INFO: found job 16884956 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:20 proc_lsst.multi:146 [INFO] found job 16884957 in provider astro proc_lsst.multi INFO: found job 16884957 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:20 proc_lsst.multi:146 [INFO] found job 16884958 in provider astro proc_lsst.multi INFO: found job 16884958 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:21 proc_lsst.multi:146 [INFO] found job 16884959 in provider astro proc_lsst.multi INFO: found job 16884959 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:23 proc_lsst.multi:146 [INFO] found job 16884960 in provider astro proc_lsst.multi INFO: found job 16884960 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:23 proc_lsst.multi:146 [INFO] found job 16884961 in provider astro proc_lsst.multi INFO: found job 16884961 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:24 proc_lsst.multi:146 [INFO] found job 16884962 in provider astro proc_lsst.multi INFO: found job 16884962 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:24 proc_lsst.multi:146 [INFO] found job 16884963 in provider astro proc_lsst.multi INFO: found job 16884963 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:24 proc_lsst.multi:146 [INFO] found job 16884964 in provider astro proc_lsst.multi INFO: found job 16884964 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:25 proc_lsst.multi:146 [INFO] found job 16884965 in provider astro proc_lsst.multi INFO: found job 16884965 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:25 proc_lsst.multi:146 [INFO] found job 16884966 in provider astro proc_lsst.multi INFO: found job 16884966 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:25 proc_lsst.multi:146 [INFO] found job 16884967 in provider astro proc_lsst.multi INFO: found job 16884967 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c766664765259d0605a05ed522fe82e2 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/65f6e188-34bd-4409-8564-f279c1b87236_isrForCrosstalkSources_1124444_9.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/65f6e188-34bd-4409-8564-f279c1b87236_isrForCrosstalkSources_1124444_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:26 proc_lsst.multi:146 [INFO] found job 16884968 in provider astro proc_lsst.multi INFO: found job 16884968 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eec8e7e9fd43de25fda70589ae34fec9 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0453c7aa-f480-44fa-8458-d3fc0e6fb4d1_isrForCrosstalkSources_1124444_30.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0453c7aa-f480-44fa-8458-d3fc0e6fb4d1_isrForCrosstalkSources_1124444_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:26 proc_lsst.multi:146 [INFO] found job 16884972 in provider astro proc_lsst.multi INFO: found job 16884972 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:26 proc_lsst.multi:146 [INFO] found job 16884973 in provider astro proc_lsst.multi INFO: found job 16884973 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:26 proc_lsst.multi:146 [INFO] found job 16884974 in provider astro proc_lsst.multi INFO: found job 16884974 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:27 proc_lsst.multi:146 [INFO] found job 16884975 in provider astro proc_lsst.multi INFO: found job 16884975 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:27 proc_lsst.multi:146 [INFO] found job 16884976 in provider astro proc_lsst.multi INFO: found job 16884976 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:27 proc_lsst.multi:146 [INFO] found job 16884977 in provider astro proc_lsst.multi INFO: found job 16884977 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:28 proc_lsst.multi:146 [INFO] found job 16884978 in provider astro proc_lsst.multi INFO: found job 16884978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:28 proc_lsst.multi:146 [INFO] found job 16884979 in provider astro proc_lsst.multi INFO: found job 16884979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:28 proc_lsst.multi:146 [INFO] found job 16884980 in provider astro proc_lsst.multi INFO: found job 16884980 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:29 proc_lsst.multi:146 [INFO] found job 16884981 in provider astro proc_lsst.multi INFO: found job 16884981 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:29 proc_lsst.multi:146 [INFO] found job 16884982 in provider astro proc_lsst.multi INFO: found job 16884982 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:29 proc_lsst.multi:146 [INFO] found job 16884983 in provider astro proc_lsst.multi INFO: found job 16884983 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:29 proc_lsst.multi:146 [INFO] found job 16884984 in provider astro proc_lsst.multi INFO: found job 16884984 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:30 proc_lsst.multi:146 [INFO] found job 16884985 in provider astro proc_lsst.multi INFO: found job 16884985 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:30 proc_lsst.multi:146 [INFO] found job 16884986 in provider astro proc_lsst.multi INFO: found job 16884986 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 114 parsl.dataflow.strategy DEBUG: Executor multi has 114 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 114 parsl.dataflow.strategy DEBUG: Executor multi has 114 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c8b623f1b46ffb2607598983cd29c8f with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b3668a60-ad2b-44ff-a9fe-ca33150eae8c_isrForCrosstalkSources_1124444_13.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b3668a60-ad2b-44ff-a9fe-ca33150eae8c_isrForCrosstalkSources_1124444_13.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e7d62590dda0dc8f2c33e4ecd941c2c with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/407980b2-e881-4916-a991-7b6cdfb04ded_isrForCrosstalkSources_1124442_22.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/407980b2-e881-4916-a991-7b6cdfb04ded_isrForCrosstalkSources_1124442_22.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 112 parsl.dataflow.strategy DEBUG: Executor multi has 112 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 112 parsl.dataflow.strategy DEBUG: Executor multi has 112 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3961c1e2068d739b04d1552e446f046d with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/3439ab4a-a3d9-4ff2-a485-5250906ee718_isrForCrosstalkSources_1124444_28.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/3439ab4a-a3d9-4ff2-a485-5250906ee718_isrForCrosstalkSources_1124444_28.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fd804843eef17d750cd28649996e95d with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/f170cbcf-2a73-4e8d-8f09-56168f358616_isrForCrosstalkSources_1124442_15.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/f170cbcf-2a73-4e8d-8f09-56168f358616_isrForCrosstalkSources_1124442_15.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bc3618d672cc13bca5f9fd76b78958e with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0624f8f3-db97-4f78-8ce9-58b7a474705c_isrForCrosstalkSources_1124444_56.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0624f8f3-db97-4f78-8ce9-58b7a474705c_isrForCrosstalkSources_1124444_56.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7fa9dc54f5e39a806f221c452b1147f with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/aa76a740-2eef-46fd-a567-2e42d3cf162b_isrForCrosstalkSources_1124444_19.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/aa76a740-2eef-46fd-a567-2e42d3cf162b_isrForCrosstalkSources_1124444_19.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 523c6e42321171afa5bf6fe1a91e5380 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/e02e5020-d19f-466f-9868-56370873c664_isrForCrosstalkSources_1124442_34.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/e02e5020-d19f-466f-9868-56370873c664_isrForCrosstalkSources_1124442_34.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 851ec338672b1fbf56543e9f94f183ad with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/217f7626-edb2-4b48-b9b3-4711b97cf40d_isrForCrosstalkSources_1124444_21.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/217f7626-edb2-4b48-b9b3-4711b97cf40d_isrForCrosstalkSources_1124444_21.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 291904314154669a02a0941a5d9f1626 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/47347ce5-e3ab-4934-bd71-6b283c754ac2_isrForCrosstalkSources_1124442_24.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/47347ce5-e3ab-4934-bd71-6b283c754ac2_isrForCrosstalkSources_1124442_24.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 139a6e5a10c9301ce62e28d8e6911502 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d291b958-f9f5-46d2-bb02-532d3533f78d_isrForCrosstalkSources_1124442_45.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d291b958-f9f5-46d2-bb02-532d3533f78d_isrForCrosstalkSources_1124442_45.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85f40cef902bcf6a27e7aa7553cf9694 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9e64bb39-0b7c-4081-8f13-2595cf97cfdb_isrForCrosstalkSources_1124444_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9e64bb39-0b7c-4081-8f13-2595cf97cfdb_isrForCrosstalkSources_1124444_27.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7f973df3b13918bf5ffaf29de0706ab with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/fb4613c3-fc04-4d7b-86dd-c155b398801b_isrForCrosstalkSources_1124442_5.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/fb4613c3-fc04-4d7b-86dd-c155b398801b_isrForCrosstalkSources_1124442_5.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bea0ee7ded39dd9f91f20dde284e3475 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/7d72c345-627e-42ae-a883-fd5ca4d8eacc_isrForCrosstalkSources_1124444_14.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/7d72c345-627e-42ae-a883-fd5ca4d8eacc_isrForCrosstalkSources_1124444_14.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21c92c13366e3ceb1718749a22526bef with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/1989d508-d417-43f8-876f-e6baa36eb653_isrForCrosstalkSources_1124442_16.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/1989d508-d417-43f8-876f-e6baa36eb653_isrForCrosstalkSources_1124442_16.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d851e52667b15ebd61916da991381cdc with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/8bd0070d-cead-49f1-b944-455fe575d88e_isrForCrosstalkSources_1124442_33.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/8bd0070d-cead-49f1-b944-455fe575d88e_isrForCrosstalkSources_1124442_33.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ace7cb8f194289f328de82e93207eaf1 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d087f34f-bcf0-4b87-9ea3-b1596a7369e9_isrForCrosstalkSources_1124442_19.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d087f34f-bcf0-4b87-9ea3-b1596a7369e9_isrForCrosstalkSources_1124442_19.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b3ec9b5062fb16d760ee3925e32e6d9 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/091dc206-8788-4056-9e81-92b9efe6c3ce_isrForCrosstalkSources_1124444_8.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/091dc206-8788-4056-9e81-92b9efe6c3ce_isrForCrosstalkSources_1124444_8.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45e0ee936ed3ad4a73b60599099bbf78 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/36248fc5-8fa5-47cf-b9fa-83a268026ffe_isrForCrosstalkSources_1124442_32.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/36248fc5-8fa5-47cf-b9fa-83a268026ffe_isrForCrosstalkSources_1124442_32.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fb983bc77626c88fa3c4a493da2315c with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/17de3b51-45c1-4b4b-b14c-2a70be1fce1b_isrForCrosstalkSources_1124444_1.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/17de3b51-45c1-4b4b-b14c-2a70be1fce1b_isrForCrosstalkSources_1124444_1.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cc3280c3c6bf5c53897798eeb791a43 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/608cdfb7-9368-4fd4-b7c0-3477c8c2d790_isrForCrosstalkSources_1124444_15.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/608cdfb7-9368-4fd4-b7c0-3477c8c2d790_isrForCrosstalkSources_1124444_15.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f73cef15f90035ab035f85ff233d2e3a with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/80458c4d-e813-4703-876e-67fe4964c7ee_isrForCrosstalkSources_1124444_18.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/80458c4d-e813-4703-876e-67fe4964c7ee_isrForCrosstalkSources_1124444_18.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e681589af10a8eedab2ec5c1f90e5908 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c3382ede-885e-44c6-85e6-d70bf05ba1fb_isrForCrosstalkSources_1124444_41.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c3382ede-885e-44c6-85e6-d70bf05ba1fb_isrForCrosstalkSources_1124444_41.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bf9b8c7616ad001223f81b5d7a5e61f with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/a38fc88a-4421-4899-a743-ff636c9a0761_isrForCrosstalkSources_1124444_7.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/a38fc88a-4421-4899-a743-ff636c9a0761_isrForCrosstalkSources_1124444_7.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 500bc466f42879f7eeccfaacb26bff41 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d2940c25-e782-457d-9956-9999eb8c6e39_isrForCrosstalkSources_1124442_3.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d2940c25-e782-457d-9956-9999eb8c6e39_isrForCrosstalkSources_1124442_3.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b03f906d60b963464ce569d15c632fe7 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e2d0722e-5837-4bcd-8ac3-b7edd109818d_isrForCrosstalkSources_1124444_3.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e2d0722e-5837-4bcd-8ac3-b7edd109818d_isrForCrosstalkSources_1124444_3.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee77b677850c014bba88e144616f0e13 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3c01704b-1a47-465e-ba1a-b1fdaef86984_isrForCrosstalkSources_1124442_30.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3c01704b-1a47-465e-ba1a-b1fdaef86984_isrForCrosstalkSources_1124442_30.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6fec459b9d422260b0b88fb023465a9 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/63f647aa-82e2-4e5b-84ba-739f3d4b4736_isrForCrosstalkSources_1124442_38.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/63f647aa-82e2-4e5b-84ba-739f3d4b4736_isrForCrosstalkSources_1124442_38.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2899dcf2310f4830847e1e0d6b64eb44 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9566eedd-c57c-4b4e-ac88-44d9b6fd2180_isrForCrosstalkSources_1124444_36.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9566eedd-c57c-4b4e-ac88-44d9b6fd2180_isrForCrosstalkSources_1124444_36.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c416cfcb4c18d8566fdd04940bfae0d2 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5d53911e-138d-47bf-9c29-31acfd87dc66_isrForCrosstalkSources_1124442_53.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5d53911e-138d-47bf-9c29-31acfd87dc66_isrForCrosstalkSources_1124442_53.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aef30bceb3d9e18ca4264b9b12cea197 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/110a6e77-f243-48b1-b938-698da46f98f3_isrForCrosstalkSources_1124444_43.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/110a6e77-f243-48b1-b938-698da46f98f3_isrForCrosstalkSources_1124444_43.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57e60162693320f892fddd5ddc68b329 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/61a3aa32-c447-4381-8a17-d2338325f5fc_isrForCrosstalkSources_1124442_7.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/61a3aa32-c447-4381-8a17-d2338325f5fc_isrForCrosstalkSources_1124442_7.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df5a2c01bcf1090a81d1f86aa7f4f1d2 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/9686d9b1-4892-48f3-bd82-c38e1170d8c3_isrForCrosstalkSources_1124442_60.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/9686d9b1-4892-48f3-bd82-c38e1170d8c3_isrForCrosstalkSources_1124442_60.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5efa4136e442f355f8aca5e8b2a8d37 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/05ce85ce-7372-4c04-8274-73ff29874238_isrForCrosstalkSources_1124442_59.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/05ce85ce-7372-4c04-8274-73ff29874238_isrForCrosstalkSources_1124442_59.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc51e1e39eead9d2802f15b4467dc745 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b0743336-27fe-46c6-b71a-746f9c4cb429_isrForCrosstalkSources_1124444_59.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/b0743336-27fe-46c6-b71a-746f9c4cb429_isrForCrosstalkSources_1124444_59.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 80 parsl.dataflow.strategy DEBUG: Executor multi has 80 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2eec9dfbfa457efd296e4b27c01879e with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/6529552a-7251-4ade-b78f-c8c3831de7af_isrForCrosstalkSources_1124444_62.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/6529552a-7251-4ade-b78f-c8c3831de7af_isrForCrosstalkSources_1124444_62.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87c80219f267bc178735581074eb7337 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/033029c5-80ce-4775-8b7b-926d45827b13_isrForCrosstalkSources_1124442_6.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/033029c5-80ce-4775-8b7b-926d45827b13_isrForCrosstalkSources_1124442_6.stderr 2024-03-07 04:11:50 proc_lsst.multi:146 [INFO] found job 59085 in provider local proc_lsst.multi INFO: found job 59085 in provider local 2024-03-07 04:11:50 proc_lsst.multi:146 [INFO] found job 16884954 in provider astro proc_lsst.multi INFO: found job 16884954 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:51 proc_lsst.multi:146 [INFO] found job 16884955 in provider astro proc_lsst.multi INFO: found job 16884955 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:51 proc_lsst.multi:146 [INFO] found job 16884956 in provider astro proc_lsst.multi INFO: found job 16884956 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:51 proc_lsst.multi:146 [INFO] found job 16884957 in provider astro proc_lsst.multi INFO: found job 16884957 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:52 proc_lsst.multi:146 [INFO] found job 16884958 in provider astro proc_lsst.multi INFO: found job 16884958 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:52 proc_lsst.multi:146 [INFO] found job 16884959 in provider astro proc_lsst.multi INFO: found job 16884959 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:52 proc_lsst.multi:146 [INFO] found job 16884960 in provider astro proc_lsst.multi INFO: found job 16884960 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:53 proc_lsst.multi:146 [INFO] found job 16884961 in provider astro proc_lsst.multi INFO: found job 16884961 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:53 proc_lsst.multi:146 [INFO] found job 16884962 in provider astro proc_lsst.multi INFO: found job 16884962 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:53 proc_lsst.multi:146 [INFO] found job 16884963 in provider astro proc_lsst.multi INFO: found job 16884963 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36f3bfa1e6632f014ae2b059b596878d with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/4283937f-7a04-44be-8262-a7842fa502fd_isrForCrosstalkSources_1124442_39.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/4283937f-7a04-44be-8262-a7842fa502fd_isrForCrosstalkSources_1124442_39.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3661abeaa55e71ec38e6b7d0a256f0bc with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c1187ceb-2f79-4ec3-af7c-634cbd61b481_isrForCrosstalkSources_1124444_40.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c1187ceb-2f79-4ec3-af7c-634cbd61b481_isrForCrosstalkSources_1124444_40.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 876e3e10f1439f8aa6ebe7c4a5adb43d with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/595d2f09-7c62-4729-93f2-e19f1fa84f84_isrForCrosstalkSources_1124444_39.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/595d2f09-7c62-4729-93f2-e19f1fa84f84_isrForCrosstalkSources_1124444_39.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab7bd91a9f49b5d455e6e015e4229119 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/c67278a2-c331-437a-bd7c-258c0e80d4f8_isrForCrosstalkSources_1124442_23.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/c67278a2-c331-437a-bd7c-258c0e80d4f8_isrForCrosstalkSources_1124442_23.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e35579c1d54d299eaf2d44d6085f0409 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0b0fa0a5-b414-426b-be5b-9806113f0dae_isrForCrosstalkSources_1124444_20.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0b0fa0a5-b414-426b-be5b-9806113f0dae_isrForCrosstalkSources_1124444_20.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edb89f612eb94d7870a9ba0461aef951 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/10c111b8-36ef-4714-9759-81449226ae37_isrForCrosstalkSources_1124444_24.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/10c111b8-36ef-4714-9759-81449226ae37_isrForCrosstalkSources_1124444_24.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86351204f95fe6787cb6347ed0c6b776 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/e5af3728-598e-438c-8297-fbf5dc29827f_isrForCrosstalkSources_1124442_13.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/e5af3728-598e-438c-8297-fbf5dc29827f_isrForCrosstalkSources_1124442_13.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31c2ceb1ac051a18fc994939ae0f3010 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/0b26a447-671b-405e-b6cc-e788b0299c0b_isrForCrosstalkSources_1124442_21.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/0b26a447-671b-405e-b6cc-e788b0299c0b_isrForCrosstalkSources_1124442_21.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb135d87ef9a6fccc16caa68adac24c6 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/64a3f1e4-f3ce-4903-9371-8ca3d26576ac_isrForCrosstalkSources_1124442_28.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/64a3f1e4-f3ce-4903-9371-8ca3d26576ac_isrForCrosstalkSources_1124442_28.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71b7018050f5bbaddc0726d8117d6bec with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/de816d44-6691-4376-acd3-cf5904f19d09_isrForCrosstalkSources_1124442_18.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/de816d44-6691-4376-acd3-cf5904f19d09_isrForCrosstalkSources_1124442_18.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 918494800ec8ffb42efeb01e0e95b393 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d034c157-fd4e-4b03-94c7-224ffb51107f_isrForCrosstalkSources_1124444_51.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d034c157-fd4e-4b03-94c7-224ffb51107f_isrForCrosstalkSources_1124444_51.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4a563a574003e73b5e4fe8c207389a7 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6f967276-b778-4fb6-9590-29cbf816d379_isrForCrosstalkSources_1124442_56.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6f967276-b778-4fb6-9590-29cbf816d379_isrForCrosstalkSources_1124442_56.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce8f0bb389e3192b25890bc7188cb75b with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6bf85c17-0032-44e9-a786-212d01195984_isrForCrosstalkSources_1124442_50.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6bf85c17-0032-44e9-a786-212d01195984_isrForCrosstalkSources_1124442_50.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee043d6161b32dbe9337ae4e35473aad with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/145d2e8b-a2d3-4cb6-91cc-525f16877f88_isrForCrosstalkSources_1124442_17.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/145d2e8b-a2d3-4cb6-91cc-525f16877f88_isrForCrosstalkSources_1124442_17.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 341426370b51fe6a7552116bc9895fe7 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ac87da8c-20a0-472a-8c48-34dafb7e87b7_isrForCrosstalkSources_1124444_23.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ac87da8c-20a0-472a-8c48-34dafb7e87b7_isrForCrosstalkSources_1124444_23.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c011be69ec6e42b0be6520b5de82037 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3813a944-a7f4-4fbb-b0bb-0b821388def7_isrForCrosstalkSources_1124442_48.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3813a944-a7f4-4fbb-b0bb-0b821388def7_isrForCrosstalkSources_1124442_48.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94df266d263c4de93152e350a71b30f2 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/0f48fa49-5c76-46dc-b9f2-958a74ec381d_isrForCrosstalkSources_1124442_41.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/0f48fa49-5c76-46dc-b9f2-958a74ec381d_isrForCrosstalkSources_1124442_41.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f16e6cef698724c5cae47b9eaf3fc5ae with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5c207647-56ce-430d-9964-1839e87e5e7b_isrForCrosstalkSources_1124442_9.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5c207647-56ce-430d-9964-1839e87e5e7b_isrForCrosstalkSources_1124442_9.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65a0a8988020ed4578f4630d631fce71 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5d646156-bf97-4902-9c63-ed837b6140cd_isrForCrosstalkSources_1124442_10.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5d646156-bf97-4902-9c63-ed837b6140cd_isrForCrosstalkSources_1124442_10.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 839ac9937a8ee28f6098f06764f824d1 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/cae58037-6795-45c0-acc9-bddeeb327109_isrForCrosstalkSources_1124444_5.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/cae58037-6795-45c0-acc9-bddeeb327109_isrForCrosstalkSources_1124444_5.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9267b6a478736dee7bfb13b7d0b7101e with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/6e16cc32-b8de-48c3-adf7-082ec399aab6_isrForCrosstalkSources_1124444_12.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/6e16cc32-b8de-48c3-adf7-082ec399aab6_isrForCrosstalkSources_1124444_12.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8fe019b6fc52789de1df7cb040fccc6 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/029fc1d3-9407-4ebe-8483-038bec0f511b_isrForCrosstalkSources_1124442_43.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/029fc1d3-9407-4ebe-8483-038bec0f511b_isrForCrosstalkSources_1124442_43.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 479cdd9899f7af0b36385e94f28e8ca1 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3ce045db-0d60-4273-84d6-d5a0513af790_isrForCrosstalkSources_1124442_44.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3ce045db-0d60-4273-84d6-d5a0513af790_isrForCrosstalkSources_1124442_44.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3a8bf7dc61f2af36d385545fab8dc54 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/8503dcd0-c5e0-432d-a9be-b7a51728a337_isrForCrosstalkSources_1124442_52.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/8503dcd0-c5e0-432d-a9be-b7a51728a337_isrForCrosstalkSources_1124442_52.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d28e4dc7ed2da9164e48e45b02a3b9de with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/a2de857e-016f-4e59-8d10-71c340e3d63b_isrForCrosstalkSources_1124442_54.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/a2de857e-016f-4e59-8d10-71c340e3d63b_isrForCrosstalkSources_1124442_54.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4a2fda805c04d4a1a9fb84eafbb3833 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/1347c6ba-f739-4bbd-834c-3ef9d1e6fefd_isrForCrosstalkSources_1124444_58.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/1347c6ba-f739-4bbd-834c-3ef9d1e6fefd_isrForCrosstalkSources_1124444_58.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdebde54d68b1e8dcfc4591e8562edda with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0a55c0f4-ebec-4226-ae0e-3cb014573df5_isrForCrosstalkSources_1124444_44.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0a55c0f4-ebec-4226-ae0e-3cb014573df5_isrForCrosstalkSources_1124444_44.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf32a9bbc0e7638f815fcd8cb4d6d7ac with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/18e8b1e9-883e-481c-8292-664fe15efe2a_isrForCrosstalkSources_1124444_31.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/18e8b1e9-883e-481c-8292-664fe15efe2a_isrForCrosstalkSources_1124444_31.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73618202eec041b9a521fe155c113da7 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/a0fe2fd9-3564-40a3-9777-1b91c657e3a2_isrForCrosstalkSources_1124442_57.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/a0fe2fd9-3564-40a3-9777-1b91c657e3a2_isrForCrosstalkSources_1124442_57.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5def5013aaad238adf1cec95bfd1b3d with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/95071f74-7345-467c-9b3c-658a5b211b45_isrForCrosstalkSources_1124444_34.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/95071f74-7345-467c-9b3c-658a5b211b45_isrForCrosstalkSources_1124444_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:55 proc_lsst.multi:146 [INFO] found job 16884964 in provider astro proc_lsst.multi INFO: found job 16884964 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:56 proc_lsst.multi:146 [INFO] found job 16884965 in provider astro proc_lsst.multi INFO: found job 16884965 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:56 proc_lsst.multi:146 [INFO] found job 16884966 in provider astro proc_lsst.multi INFO: found job 16884966 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:56 proc_lsst.multi:146 [INFO] found job 16884967 in provider astro proc_lsst.multi INFO: found job 16884967 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:57 proc_lsst.multi:146 [INFO] found job 16884968 in provider astro proc_lsst.multi INFO: found job 16884968 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:57 proc_lsst.multi:146 [INFO] found job 16884972 in provider astro proc_lsst.multi INFO: found job 16884972 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46f02d97794be9f5e7cca65b1bb347e4 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/20547dae-7e85-4c33-932c-2348183191ac_isrForCrosstalkSources_1124442_35.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/20547dae-7e85-4c33-932c-2348183191ac_isrForCrosstalkSources_1124442_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:57 proc_lsst.multi:146 [INFO] found job 16884973 in provider astro proc_lsst.multi INFO: found job 16884973 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:58 proc_lsst.multi:146 [INFO] found job 16884974 in provider astro proc_lsst.multi INFO: found job 16884974 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:58 proc_lsst.multi:146 [INFO] found job 16884975 in provider astro proc_lsst.multi INFO: found job 16884975 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2a54c0671854e858ea2ace51e6a6049 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/bbfc14ff-1c31-4c8d-87ff-d412ddcc18b5_isrForCrosstalkSources_1124442_27.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/bbfc14ff-1c31-4c8d-87ff-d412ddcc18b5_isrForCrosstalkSources_1124442_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:58 proc_lsst.multi:146 [INFO] found job 16884976 in provider astro proc_lsst.multi INFO: found job 16884976 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:58 proc_lsst.multi:146 [INFO] found job 16884977 in provider astro proc_lsst.multi INFO: found job 16884977 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:59 proc_lsst.multi:146 [INFO] found job 16884978 in provider astro proc_lsst.multi INFO: found job 16884978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:59 proc_lsst.multi:146 [INFO] found job 16884979 in provider astro proc_lsst.multi INFO: found job 16884979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:11:59 proc_lsst.multi:146 [INFO] found job 16884980 in provider astro proc_lsst.multi INFO: found job 16884980 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:12:00 proc_lsst.multi:146 [INFO] found job 16884981 in provider astro proc_lsst.multi INFO: found job 16884981 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:12:00 proc_lsst.multi:146 [INFO] found job 16884982 in provider astro proc_lsst.multi INFO: found job 16884982 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:12:01 proc_lsst.multi:146 [INFO] found job 16884983 in provider astro proc_lsst.multi INFO: found job 16884983 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:12:01 proc_lsst.multi:146 [INFO] found job 16884984 in provider astro proc_lsst.multi INFO: found job 16884984 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:12:01 proc_lsst.multi:146 [INFO] found job 16884985 in provider astro proc_lsst.multi INFO: found job 16884985 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING 2024-03-07 04:12:02 proc_lsst.multi:146 [INFO] found job 16884986 in provider astro proc_lsst.multi INFO: found job 16884986 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884954,16884955,16884956,16884957,16884958,16884959,16884960,16884961,16884962,16884963,16884964,16884965,16884966,16884967,16884968,16884972,16884973,16884974,16884975,16884976,16884977,16884978,16884979,16884980,16884981,16884982,16884983,16884984,16884985,16884986' parsl.providers.slurm.slurm DEBUG: squeue returned 16884983 R 16884984 R 16884985 R 16884986 R 16884973 R 16884974 R 16884975 R 16884976 R 16884977 R 16884978 R 16884979 R 16884980 R 16884981 R 16884982 R 16884972 R 16884966 R 16884967 R 16884968 R 16884960 R 16884961 R 16884962 R 16884963 R 16884964 R 16884965 R 16884955 R 16884956 R 16884957 R 16884958 R 16884959 R 16884954 R parsl.providers.slurm.slurm DEBUG: Updating job 16884983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884972 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884956 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884954 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 46 parsl.dataflow.strategy DEBUG: Executor multi has 46 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 46 parsl.dataflow.strategy DEBUG: Executor multi has 46 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7360c15e42fc09d8709a2072e164f8b5 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/697ac075-8814-43f3-abc4-2a66df22e1a6_isrForCrosstalkSources_1124442_20.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/697ac075-8814-43f3-abc4-2a66df22e1a6_isrForCrosstalkSources_1124442_20.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1eba89cb3d5aa600b0d5bf47cc8f27e1 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/494fb10d-23f5-4539-a255-881a002bb7ab_isrForCrosstalkSources_1124444_16.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/494fb10d-23f5-4539-a255-881a002bb7ab_isrForCrosstalkSources_1124444_16.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13fb2580a4a14884519bff76846f52cf with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/9e7e54bf-1351-4cb3-9a84-2fe97dca51b2_isrForCrosstalkSources_1124442_1.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/9e7e54bf-1351-4cb3-9a84-2fe97dca51b2_isrForCrosstalkSources_1124442_1.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 483f28f4609527fff4c0b8603aa11420 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/1ad6ac9e-8a08-43c6-9740-26b63218c714_isrForCrosstalkSources_1124442_49.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/1ad6ac9e-8a08-43c6-9740-26b63218c714_isrForCrosstalkSources_1124442_49.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17dd1d15a64fc49e1eb0cfff30b1fdc9 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/eaa4379d-b15c-4b5c-988f-e8737d7eac36_isrForCrosstalkSources_1124444_2.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/eaa4379d-b15c-4b5c-988f-e8737d7eac36_isrForCrosstalkSources_1124444_2.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 966c8dbe4bd62d5240be7a842bd00f1e with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ed07a291-b2a6-4029-90f0-028f737ad5dc_isrForCrosstalkSources_1124442_37.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ed07a291-b2a6-4029-90f0-028f737ad5dc_isrForCrosstalkSources_1124442_37.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97834d10a7157f5d02b5005dad979592 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/73e42271-1adc-4d34-a1d9-8cf8f31e4f95_isrForCrosstalkSources_1124444_50.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/73e42271-1adc-4d34-a1d9-8cf8f31e4f95_isrForCrosstalkSources_1124444_50.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2c549166ac9188f5b5a0a290dd2f520 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/8c0ec435-1811-4ffb-8a8e-c8b7f94283bb_isrForCrosstalkSources_1124444_52.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/8c0ec435-1811-4ffb-8a8e-c8b7f94283bb_isrForCrosstalkSources_1124444_52.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 075d7ed12b2fd14b5b07b266801818d1 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/5763aea6-8d3a-4619-8355-27efc6a2d07e_isrForCrosstalkSources_1124444_48.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/5763aea6-8d3a-4619-8355-27efc6a2d07e_isrForCrosstalkSources_1124444_48.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f50f398cfea855eeabb69fb48bef8ef2 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/a95038da-bbe8-473c-beb0-50f2bd069384_isrForCrosstalkSources_1124444_53.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/a95038da-bbe8-473c-beb0-50f2bd069384_isrForCrosstalkSources_1124444_53.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9be50ba36de379b1b7877cd04acf62ac with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/50051efb-806f-4f18-869d-8ccf9316d891_isrForCrosstalkSources_1124444_45.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/50051efb-806f-4f18-869d-8ccf9316d891_isrForCrosstalkSources_1124444_45.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f50c37036ed2d70c94bc40ab76843b21 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/589994e1-455c-4c34-8bb6-45bff732c5a5_isrForCrosstalkSources_1124442_8.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/589994e1-455c-4c34-8bb6-45bff732c5a5_isrForCrosstalkSources_1124442_8.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce8f53fcda7c570de4cac96318cf389b with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/4844f0af-6d63-4286-9ce9-af824883d189_isrForCrosstalkSources_1124442_36.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/4844f0af-6d63-4286-9ce9-af824883d189_isrForCrosstalkSources_1124442_36.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43fe3f0666b7d7d1599675eb7b89bdd2 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/93770227-ae55-403d-bb78-2cef5ee49b86_isrForCrosstalkSources_1124444_25.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/93770227-ae55-403d-bb78-2cef5ee49b86_isrForCrosstalkSources_1124444_25.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7aca312294584ad59852e98f84cb5694 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/2e23dc98-e685-4675-b1fc-c9e4837581b1_isrForCrosstalkSources_1124442_46.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/2e23dc98-e685-4675-b1fc-c9e4837581b1_isrForCrosstalkSources_1124442_46.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96f8f81032256e12a7303794715dbe81 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/42de30c1-d7a4-4f71-a27e-5b7ed97163c3_isrForCrosstalkSources_1124442_31.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/42de30c1-d7a4-4f71-a27e-5b7ed97163c3_isrForCrosstalkSources_1124442_31.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8334f05648cfffe411a49784bba9946 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9e95e977-5049-4232-af3e-07bfa8cc25d8_isrForCrosstalkSources_1124444_10.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/9e95e977-5049-4232-af3e-07bfa8cc25d8_isrForCrosstalkSources_1124444_10.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4110d691aff7a265dc09996da681f53 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/61aa40c0-c887-4d2f-a0c4-c0483290f63c_isrForCrosstalkSources_1124444_46.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/61aa40c0-c887-4d2f-a0c4-c0483290f63c_isrForCrosstalkSources_1124444_46.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb56373f269ab5bb341232b9c60b7026 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/be8a98d0-aeb2-4709-a5e8-cb687cc24f0d_isrForCrosstalkSources_1124442_61.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/be8a98d0-aeb2-4709-a5e8-cb687cc24f0d_isrForCrosstalkSources_1124442_61.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd74bd8e3ccad951709d3e340eef8acc with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/7486d1b5-5c1e-4381-b7c7-45f58044e36b_isrForCrosstalkSources_1124442_40.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/7486d1b5-5c1e-4381-b7c7-45f58044e36b_isrForCrosstalkSources_1124442_40.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48a0db2c42a7bc940094c777d98beb8b with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/b9931ca4-dcd7-402e-98db-88bc9da7b3bc_isrForCrosstalkSources_1124442_58.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/b9931ca4-dcd7-402e-98db-88bc9da7b3bc_isrForCrosstalkSources_1124442_58.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0386c39292040cdf31827b4f4bd9e3cf with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/69b5478d-3cdc-40a3-9baf-acd091560a33_isrForCrosstalkSources_1124442_42.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/69b5478d-3cdc-40a3-9baf-acd091560a33_isrForCrosstalkSources_1124442_42.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a660fc4e09e4b8a2fd3542f88082ef48 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0cec5cac-3914-4286-b1c1-0438125e9f4d_isrForCrosstalkSources_1124444_47.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0cec5cac-3914-4286-b1c1-0438125e9f4d_isrForCrosstalkSources_1124444_47.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21ab8b2e97bef11fb1ec46794c78201b with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ddafa3b0-8369-4af8-b6e1-f9bc9e36d9b5_isrForCrosstalkSources_1124444_11.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ddafa3b0-8369-4af8-b6e1-f9bc9e36d9b5_isrForCrosstalkSources_1124444_11.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e9549fcbd2df8546369b9b752b565ea with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0d2c722e-ffd5-498b-aff6-2daad731436e_isrForCrosstalkSources_1124444_54.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/0d2c722e-ffd5-498b-aff6-2daad731436e_isrForCrosstalkSources_1124444_54.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4033ebaaa7ea4bcee31af354b8801854 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d8c645f2-5772-4392-b031-d9244f75c4df_isrForCrosstalkSources_1124442_62.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d8c645f2-5772-4392-b031-d9244f75c4df_isrForCrosstalkSources_1124442_62.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f645c57df7d691cbb1772c9f386f49f with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c6ef7975-6e77-4a55-b74d-eaf5da584ccc_isrForCrosstalkSources_1124444_57.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/c6ef7975-6e77-4a55-b74d-eaf5da584ccc_isrForCrosstalkSources_1124444_57.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12af00c17c3b84c162fa698bb16fcd8e with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/63727be1-3c1b-49fa-b917-98f5c8bd5bdb_isrForCrosstalkSources_1124444_29.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/63727be1-3c1b-49fa-b917-98f5c8bd5bdb_isrForCrosstalkSources_1124444_29.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 006abf90395beb78203afbdc626499b2 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/7eb3e0ec-1230-4069-af56-be8057d10437_isrForCrosstalkSources_1124444_61.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/7eb3e0ec-1230-4069-af56-be8057d10437_isrForCrosstalkSources_1124444_61.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 473fa49c094333442ae9edd54af168f3 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3fcd92bc-3593-4a5c-a9f2-4e2f8a2db0cb_isrForCrosstalkSources_1124442_11.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/3fcd92bc-3593-4a5c-a9f2-4e2f8a2db0cb_isrForCrosstalkSources_1124442_11.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ef09c48f93491add370d4608f865a03 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e1d117ae-37aa-4dc4-b176-aa3aaa21c170_isrForCrosstalkSources_1124444_37.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e1d117ae-37aa-4dc4-b176-aa3aaa21c170_isrForCrosstalkSources_1124444_37.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cff50c45c3ab2445d38d70f9b2c9353 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d99c318c-fc92-4676-af6f-2134918ef1ca_isrForCrosstalkSources_1124442_47.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/d99c318c-fc92-4676-af6f-2134918ef1ca_isrForCrosstalkSources_1124442_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 14 parsl.dataflow.strategy DEBUG: Executor multi has 14 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 31, active_tasks = 14 parsl.dataflow.strategy DEBUG: Executor multi has 14 active tasks, 31/0 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23024206041488 parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9693dd9e1852a8e1657ddeb75dd5ff03 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d3a01e21-dd42-46da-91f9-21f6466692e9_isrForCrosstalkSources_1124444_33.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/d3a01e21-dd42-46da-91f9-21f6466692e9_isrForCrosstalkSources_1124444_33.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 666166337c8f3b6e95e8bfa02ddd6f4c with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e5e0bc4a-f7ef-43e2-a9e9-6e8a2e9dc885_isrForCrosstalkSources_1124444_4.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e5e0bc4a-f7ef-43e2-a9e9-6e8a2e9dc885_isrForCrosstalkSources_1124444_4.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16bc9d8105340cfee6be0bf725b595f2 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/59c767ff-9529-4673-8e2f-20102917a420_isrForCrosstalkSources_1124444_35.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/59c767ff-9529-4673-8e2f-20102917a420_isrForCrosstalkSources_1124444_35.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5aa4a7658b2e53ac4ed03abb7b33800 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/68196536-94cc-41a4-8aa5-974d4052b298_isrForCrosstalkSources_1124442_4.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/68196536-94cc-41a4-8aa5-974d4052b298_isrForCrosstalkSources_1124442_4.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 258f11ed8f447828609182c0d24678f6 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6497f69f-d326-463a-912c-d896d00a27f7_isrForCrosstalkSources_1124442_14.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/6497f69f-d326-463a-912c-d896d00a27f7_isrForCrosstalkSources_1124442_14.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8ade0eb212eb53378af9a326bb09c0e with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e9980e8e-681c-4cfa-9f74-7f8238f587c3_isrForCrosstalkSources_1124444_42.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/e9980e8e-681c-4cfa-9f74-7f8238f587c3_isrForCrosstalkSources_1124444_42.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69bbcede7fe67d1c1702007185fe2813 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/8ceb7a56-193f-4b98-8dd6-546f906b36ef_isrForCrosstalkSources_1124444_55.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/8ceb7a56-193f-4b98-8dd6-546f906b36ef_isrForCrosstalkSources_1124444_55.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3d83e85f222e7ee3ef85f583862ed26 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ec04b458-3f04-4b39-aeb3-1567251461df_isrForCrosstalkSources_1124442_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/ec04b458-3f04-4b39-aeb3-1567251461df_isrForCrosstalkSources_1124442_2.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a194a5185f9f9f6c9dd0dd0441f7bf03 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5dadfcab-7174-4df7-8979-2c0a1e7ac008_isrForCrosstalkSources_1124442_12.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/5dadfcab-7174-4df7-8979-2c0a1e7ac008_isrForCrosstalkSources_1124442_12.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4895dbde1b5221ac245d3af0b777a5bb with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/2a4e6f2f-055f-4f05-a175-7d2a7d6171a5_isrForCrosstalkSources_1124442_25.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/2a4e6f2f-055f-4f05-a175-7d2a7d6171a5_isrForCrosstalkSources_1124442_25.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e5ba4112d34f6da5180a43d5f042228 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/980b148e-4bb0-417d-97fa-01d11bc387ce_isrForCrosstalkSources_1124442_29.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/980b148e-4bb0-417d-97fa-01d11bc387ce_isrForCrosstalkSources_1124442_29.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf612ed3e29491cd2d5a2c1b661322ac with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ae84afd4-a62e-4dfe-abc2-808ab7746bc4_isrForCrosstalkSources_1124444_6.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/ae84afd4-a62e-4dfe-abc2-808ab7746bc4_isrForCrosstalkSources_1124444_6.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbca39250aefc9571f5fefd2f1890cda with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/cd1b5169-d91a-4069-b3fa-5938bea4087b_isrForCrosstalkSources_1124442_55.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124442/cd1b5169-d91a-4069-b3fa-5938bea4087b_isrForCrosstalkSources_1124442_55.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0210d00f7860fd93566ec473c5e93f5d with result from task 119 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 124 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/191f2053-0ed9-4ff2-a8aa-3496ce93cc49_isrForCrosstalkSources_1124444_22.stdout parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/submit/DEEP/20220822/B1a/science#step0/20240307T041022Z/logs/isrForCrosstalkSources/1124444/191f2053-0ed9-4ff2-a8aa-3496ce93cc49_isrForCrosstalkSources_1124444_22.stderr parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=31, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 31 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 249aec079502 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 249aec079502 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5803284874cc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5803284874cc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 02f82d0a3eb3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 02f82d0a3eb3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bd0842b38441 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bd0842b38441 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b4ded42a949b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b4ded42a949b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a83e118f4192 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a83e118f4192 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 89ce1333c7bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 89ce1333c7bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4509411fdf1a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4509411fdf1a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 04103d97f9d3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 04103d97f9d3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ceeadbd6186f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ceeadbd6186f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2a708e777124 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2a708e777124 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c7bd59664c55 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c7bd59664c55 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fc779399ade7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fc779399ade7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 044c2a0a6cc4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 044c2a0a6cc4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c69e6ed28fac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c69e6ed28fac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df65304ce9b9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df65304ce9b9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f32eeeda0339 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f32eeeda0339 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 122af8f412d7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 122af8f412d7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d98a7399d2a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d98a7399d2a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: be5891eff0c9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: be5891eff0c9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 11fca01d6f15 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 11fca01d6f15 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e583900eb253 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e583900eb253 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f58b9b42cbd4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f58b9b42cbd4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b9ef259fd512 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b9ef259fd512 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3c2d6a2fd1fe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3c2d6a2fd1fe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b0eb15e5250f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b0eb15e5250f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d39872bbf9d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d39872bbf9d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 941dd2d59e7e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 941dd2d59e7e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b301aa946419 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b301aa946419 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2dc922f0b772 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2dc922f0b772 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1a2fe83fc80b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1a2fe83fc80b 2024-03-07 04:12:14 proc_lsst.multi:146 [INFO] found job 16884954 in provider astro proc_lsst.multi INFO: found job 16884954 in provider astro 2024-03-07 04:12:14 proc_lsst.multi:201 [INFO] cancelling 16884954 on provider astro proc_lsst.multi INFO: cancelling 16884954 on provider astro 2024-03-07 04:12:14 proc_lsst.multi:146 [INFO] found job 16884975 in provider astro proc_lsst.multi INFO: found job 16884975 in provider astro 2024-03-07 04:12:14 proc_lsst.multi:201 [INFO] cancelling 16884975 on provider astro proc_lsst.multi INFO: cancelling 16884975 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884956 in provider astro proc_lsst.multi INFO: found job 16884956 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884956 on provider astro proc_lsst.multi INFO: cancelling 16884956 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884961 in provider astro proc_lsst.multi INFO: found job 16884961 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884961 on provider astro proc_lsst.multi INFO: cancelling 16884961 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884957 in provider astro proc_lsst.multi INFO: found job 16884957 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884957 on provider astro proc_lsst.multi INFO: cancelling 16884957 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884962 in provider astro proc_lsst.multi INFO: found job 16884962 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884962 on provider astro proc_lsst.multi INFO: cancelling 16884962 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884981 in provider astro proc_lsst.multi INFO: found job 16884981 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884981 on provider astro proc_lsst.multi INFO: cancelling 16884981 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884960 in provider astro proc_lsst.multi INFO: found job 16884960 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884960 on provider astro proc_lsst.multi INFO: cancelling 16884960 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884963 in provider astro proc_lsst.multi INFO: found job 16884963 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884963 on provider astro proc_lsst.multi INFO: cancelling 16884963 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884973 in provider astro proc_lsst.multi INFO: found job 16884973 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884973 on provider astro proc_lsst.multi INFO: cancelling 16884973 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884958 in provider astro proc_lsst.multi INFO: found job 16884958 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884958 on provider astro proc_lsst.multi INFO: cancelling 16884958 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884959 in provider astro proc_lsst.multi INFO: found job 16884959 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884959 on provider astro proc_lsst.multi INFO: cancelling 16884959 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 16884955 in provider astro proc_lsst.multi INFO: found job 16884955 in provider astro 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 16884955 on provider astro proc_lsst.multi INFO: cancelling 16884955 on provider astro 2024-03-07 04:12:15 proc_lsst.multi:146 [INFO] found job 59085 in provider local proc_lsst.multi INFO: found job 59085 in provider local 2024-03-07 04:12:15 proc_lsst.multi:201 [INFO] cancelling 59085 on provider local proc_lsst.multi INFO: cancelling 59085 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 59085 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884967 in provider astro proc_lsst.multi INFO: found job 16884967 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884967 on provider astro proc_lsst.multi INFO: cancelling 16884967 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884966 in provider astro proc_lsst.multi INFO: found job 16884966 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884966 on provider astro proc_lsst.multi INFO: cancelling 16884966 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884965 in provider astro proc_lsst.multi INFO: found job 16884965 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884965 on provider astro proc_lsst.multi INFO: cancelling 16884965 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884972 in provider astro proc_lsst.multi INFO: found job 16884972 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884972 on provider astro proc_lsst.multi INFO: cancelling 16884972 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884980 in provider astro proc_lsst.multi INFO: found job 16884980 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884980 on provider astro proc_lsst.multi INFO: cancelling 16884980 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884983 in provider astro proc_lsst.multi INFO: found job 16884983 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884983 on provider astro proc_lsst.multi INFO: cancelling 16884983 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884976 in provider astro proc_lsst.multi INFO: found job 16884976 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884976 on provider astro proc_lsst.multi INFO: cancelling 16884976 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884985 in provider astro proc_lsst.multi INFO: found job 16884985 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884985 on provider astro proc_lsst.multi INFO: cancelling 16884985 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884968 in provider astro proc_lsst.multi INFO: found job 16884968 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884968 on provider astro proc_lsst.multi INFO: cancelling 16884968 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884982 in provider astro proc_lsst.multi INFO: found job 16884982 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884982 on provider astro proc_lsst.multi INFO: cancelling 16884982 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884977 in provider astro proc_lsst.multi INFO: found job 16884977 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884977 on provider astro proc_lsst.multi INFO: cancelling 16884977 on provider astro 2024-03-07 04:12:16 proc_lsst.multi:146 [INFO] found job 16884964 in provider astro proc_lsst.multi INFO: found job 16884964 in provider astro 2024-03-07 04:12:16 proc_lsst.multi:201 [INFO] cancelling 16884964 on provider astro proc_lsst.multi INFO: cancelling 16884964 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884979 in provider astro proc_lsst.multi INFO: found job 16884979 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884979 on provider astro proc_lsst.multi INFO: cancelling 16884979 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884978 in provider astro proc_lsst.multi INFO: found job 16884978 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884978 on provider astro proc_lsst.multi INFO: cancelling 16884978 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884986 in provider astro proc_lsst.multi INFO: found job 16884986 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884986 on provider astro proc_lsst.multi INFO: cancelling 16884986 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884984 in provider astro proc_lsst.multi INFO: found job 16884984 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884984 on provider astro proc_lsst.multi INFO: cancelling 16884984 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884974 in provider astro proc_lsst.multi INFO: found job 16884974 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884974 on provider astro proc_lsst.multi INFO: cancelling 16884974 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-07 04:12:17 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-07 04:12:17 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16884954', '16884955', '16884956', '16884957', '16884958', '16884959', '16884960', '16884961', '16884962', '16884963', '16884964', '16884965', '16884966', '16884967', '16884968', '16884972', '16884973', '16884974', '16884975', '16884976', '16884977', '16884978', '16884979', '16884980', '16884981', '16884982', '16884983', '16884984', '16884985', '16884986', '59085'] proc_lsst.multi INFO: new jobs since last cancel ['16884954', '16884955', '16884956', '16884957', '16884958', '16884959', '16884960', '16884961', '16884962', '16884963', '16884964', '16884965', '16884966', '16884967', '16884968', '16884972', '16884973', '16884974', '16884975', '16884976', '16884977', '16884978', '16884979', '16884980', '16884981', '16884982', '16884983', '16884984', '16884985', '16884986', '59085'] 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884954 in provider astro proc_lsst.multi INFO: found job 16884954 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884954 on provider astro proc_lsst.multi INFO: cancelling 16884954 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884955 in provider astro proc_lsst.multi INFO: found job 16884955 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884955 on provider astro proc_lsst.multi INFO: cancelling 16884955 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884956 in provider astro proc_lsst.multi INFO: found job 16884956 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884956 on provider astro proc_lsst.multi INFO: cancelling 16884956 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884957 in provider astro proc_lsst.multi INFO: found job 16884957 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884957 on provider astro proc_lsst.multi INFO: cancelling 16884957 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884958 in provider astro proc_lsst.multi INFO: found job 16884958 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884958 on provider astro proc_lsst.multi INFO: cancelling 16884958 on provider astro 2024-03-07 04:12:17 proc_lsst.multi:146 [INFO] found job 16884959 in provider astro proc_lsst.multi INFO: found job 16884959 in provider astro 2024-03-07 04:12:17 proc_lsst.multi:201 [INFO] cancelling 16884959 on provider astro proc_lsst.multi INFO: cancelling 16884959 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884960 in provider astro proc_lsst.multi INFO: found job 16884960 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884960 on provider astro proc_lsst.multi INFO: cancelling 16884960 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884961 in provider astro proc_lsst.multi INFO: found job 16884961 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884961 on provider astro proc_lsst.multi INFO: cancelling 16884961 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884962 in provider astro proc_lsst.multi INFO: found job 16884962 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884962 on provider astro proc_lsst.multi INFO: cancelling 16884962 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884963 in provider astro proc_lsst.multi INFO: found job 16884963 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884963 on provider astro proc_lsst.multi INFO: cancelling 16884963 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884964 in provider astro proc_lsst.multi INFO: found job 16884964 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884964 on provider astro proc_lsst.multi INFO: cancelling 16884964 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884965 in provider astro proc_lsst.multi INFO: found job 16884965 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884965 on provider astro proc_lsst.multi INFO: cancelling 16884965 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884966 in provider astro proc_lsst.multi INFO: found job 16884966 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884966 on provider astro proc_lsst.multi INFO: cancelling 16884966 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884967 in provider astro proc_lsst.multi INFO: found job 16884967 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884967 on provider astro proc_lsst.multi INFO: cancelling 16884967 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884968 in provider astro proc_lsst.multi INFO: found job 16884968 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884968 on provider astro proc_lsst.multi INFO: cancelling 16884968 on provider astro 2024-03-07 04:12:18 proc_lsst.multi:146 [INFO] found job 16884972 in provider astro proc_lsst.multi INFO: found job 16884972 in provider astro 2024-03-07 04:12:18 proc_lsst.multi:201 [INFO] cancelling 16884972 on provider astro proc_lsst.multi INFO: cancelling 16884972 on provider astro 2024-03-07 04:12:19 proc_lsst.multi:146 [INFO] found job 16884973 in provider astro proc_lsst.multi INFO: found job 16884973 in provider astro 2024-03-07 04:12:19 proc_lsst.multi:201 [INFO] cancelling 16884973 on provider astro proc_lsst.multi INFO: cancelling 16884973 on provider astro 2024-03-07 04:12:19 proc_lsst.multi:146 [INFO] found job 16884974 in provider astro proc_lsst.multi INFO: found job 16884974 in provider astro 2024-03-07 04:12:19 proc_lsst.multi:201 [INFO] cancelling 16884974 on provider astro proc_lsst.multi INFO: cancelling 16884974 on provider astro 2024-03-07 04:12:19 proc_lsst.multi:146 [INFO] found job 16884975 in provider astro proc_lsst.multi INFO: found job 16884975 in provider astro 2024-03-07 04:12:19 proc_lsst.multi:201 [INFO] cancelling 16884975 on provider astro proc_lsst.multi INFO: cancelling 16884975 on provider astro 2024-03-07 04:12:19 proc_lsst.multi:146 [INFO] found job 16884976 in provider astro proc_lsst.multi INFO: found job 16884976 in provider astro 2024-03-07 04:12:19 proc_lsst.multi:201 [INFO] cancelling 16884976 on provider astro proc_lsst.multi INFO: cancelling 16884976 on provider astro 2024-03-07 04:12:19 proc_lsst.multi:146 [INFO] found job 16884977 in provider astro proc_lsst.multi INFO: found job 16884977 in provider astro 2024-03-07 04:12:19 proc_lsst.multi:201 [INFO] cancelling 16884977 on provider astro proc_lsst.multi INFO: cancelling 16884977 on provider astro 2024-03-07 04:12:19 proc_lsst.multi:146 [INFO] found job 16884978 in provider astro proc_lsst.multi INFO: found job 16884978 in provider astro 2024-03-07 04:12:19 proc_lsst.multi:201 [INFO] cancelling 16884978 on provider astro proc_lsst.multi INFO: cancelling 16884978 on provider astro 2024-03-07 04:12:19 proc_lsst.multi:146 [INFO] found job 16884979 in provider astro proc_lsst.multi INFO: found job 16884979 in provider astro 2024-03-07 04:12:19 proc_lsst.multi:201 [INFO] cancelling 16884979 on provider astro proc_lsst.multi INFO: cancelling 16884979 on provider astro 2024-03-07 04:12:19 proc_lsst.multi:146 [INFO] found job 16884980 in provider astro proc_lsst.multi INFO: found job 16884980 in provider astro 2024-03-07 04:12:19 proc_lsst.multi:201 [INFO] cancelling 16884980 on provider astro proc_lsst.multi INFO: cancelling 16884980 on provider astro 2024-03-07 04:12:20 proc_lsst.multi:146 [INFO] found job 16884981 in provider astro proc_lsst.multi INFO: found job 16884981 in provider astro 2024-03-07 04:12:20 proc_lsst.multi:201 [INFO] cancelling 16884981 on provider astro proc_lsst.multi INFO: cancelling 16884981 on provider astro 2024-03-07 04:12:20 proc_lsst.multi:146 [INFO] found job 16884982 in provider astro proc_lsst.multi INFO: found job 16884982 in provider astro 2024-03-07 04:12:20 proc_lsst.multi:201 [INFO] cancelling 16884982 on provider astro proc_lsst.multi INFO: cancelling 16884982 on provider astro 2024-03-07 04:12:20 proc_lsst.multi:146 [INFO] found job 16884983 in provider astro proc_lsst.multi INFO: found job 16884983 in provider astro 2024-03-07 04:12:20 proc_lsst.multi:201 [INFO] cancelling 16884983 on provider astro proc_lsst.multi INFO: cancelling 16884983 on provider astro 2024-03-07 04:12:20 proc_lsst.multi:146 [INFO] found job 16884984 in provider astro proc_lsst.multi INFO: found job 16884984 in provider astro 2024-03-07 04:12:20 proc_lsst.multi:201 [INFO] cancelling 16884984 on provider astro proc_lsst.multi INFO: cancelling 16884984 on provider astro 2024-03-07 04:12:20 proc_lsst.multi:146 [INFO] found job 16884985 in provider astro proc_lsst.multi INFO: found job 16884985 in provider astro 2024-03-07 04:12:20 proc_lsst.multi:201 [INFO] cancelling 16884985 on provider astro proc_lsst.multi INFO: cancelling 16884985 on provider astro 2024-03-07 04:12:21 proc_lsst.multi:146 [INFO] found job 16884986 in provider astro proc_lsst.multi INFO: found job 16884986 in provider astro 2024-03-07 04:12:21 proc_lsst.multi:201 [INFO] cancelling 16884986 on provider astro proc_lsst.multi INFO: cancelling 16884986 on provider astro 2024-03-07 04:12:21 proc_lsst.multi:146 [INFO] found job 59085 in provider local proc_lsst.multi INFO: found job 59085 in provider local 2024-03-07 04:12:21 proc_lsst.multi:201 [INFO] cancelling 59085 on provider local proc_lsst.multi INFO: cancelling 59085 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 59085 parsl.providers.local.local WARNING: Failed to kill PID: 59085 and child processes on local 2024-03-07 04:12:22 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 108.3792 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220822_B1a_science#step0_20240307T041022Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 108.3863 seconds; current memory usage: 0.270 Gibyte, delta: -0.000 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 118.1990 seconds; current memory usage: 0.270 Gibyte, delta: 0.134 Gibyte, peak delta: 0.136 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.272 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220822_B1a_science#step0_20240307T041022Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up