+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20201017/B1c --output-run DEEP/20201017/B1c/science#step0/20240313T231242Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 2.6537 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.191 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0041 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0065 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/pipeline_orig.qgraph DEEP/20201017/B1c/science#step0/20240313T231242Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.6442 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 7.0202 seconds; current memory usage: 0.272 Gibyte, delta: 0.094 Gibyte, peak delta: 0.082 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0125 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 372 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20201017_B1c_science#step0_20240313T231242Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0817 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 374 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.1998 seconds; current memory usage: 0.274 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 99a18bb4-8e56-466e-857c-51db41d57b2f parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-13 23:13:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-13 23:13:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 23:13:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-13 23:13:06 proc_lsst.multi:170 [INFO] [multi] job_id 20175 proc_lsst.multi INFO: [multi] job_id 20175 2024-03-13 23:13:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:06 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 20175 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 20175 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 20175 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 390c1aad28145d9300ed5462ca506472 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/18ffcc0a-2566-4666-80b7-624cadcdacce_isrForCrosstalkSources_946370_29.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/18ffcc0a-2566-4666-80b7-624cadcdacce_isrForCrosstalkSources_946370_29.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 61f310fe5d9797122ffb93e103ca7143 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/27683d21-d3fe-45af-8973-2803f86a828b_isrForCrosstalkSources_946479_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/27683d21-d3fe-45af-8973-2803f86a828b_isrForCrosstalkSources_946479_59.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 659f42c433c4a1f3d42ff8331fff2d77 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/a6e19a6e-c87b-4a08-9c71-3257b6f83fa8_isrForCrosstalkSources_946477_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/a6e19a6e-c87b-4a08-9c71-3257b6f83fa8_isrForCrosstalkSources_946477_44.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash e353f533e56e2e723cce370cdcecf6c2 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e4b4c35e-b586-4a9f-a54a-8006538f9ae1_isrForCrosstalkSources_946477_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e4b4c35e-b586-4a9f-a54a-8006538f9ae1_isrForCrosstalkSources_946477_61.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash abded8e68975037d9f52517f564585cf parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/066db2ee-4e0b-493b-a6f3-8a3f943a5e38_isrForCrosstalkSources_946369_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/066db2ee-4e0b-493b-a6f3-8a3f943a5e38_isrForCrosstalkSources_946369_42.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 053f8a24ef73539fc8d3b9f0b2e734e2 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/cf5eb18e-8bf4-47c7-b862-683a58c1b9a7_isrForCrosstalkSources_946478_4.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/cf5eb18e-8bf4-47c7-b862-683a58c1b9a7_isrForCrosstalkSources_946478_4.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash e5164eab51d954dd1d99fc4c0506d942 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/17151c56-a534-48c0-a781-13091ed19211_isrForCrosstalkSources_946369_48.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/17151c56-a534-48c0-a781-13091ed19211_isrForCrosstalkSources_946369_48.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash fa4775ec808116dbdcb777482b2e91e0 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/2ef60bc2-5357-410c-993d-c42a178cf7b0_isrForCrosstalkSources_946371_48.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/2ef60bc2-5357-410c-993d-c42a178cf7b0_isrForCrosstalkSources_946371_48.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 32ac3b10411162bf222fadf1e9090b39 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c6786af6-cd2d-417c-94e1-babb04db840a_isrForCrosstalkSources_946369_36.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c6786af6-cd2d-417c-94e1-babb04db840a_isrForCrosstalkSources_946369_36.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 6b0784bdab057a7fd10da0d5de5043a2 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/2b19b237-a3a9-47ac-97c0-e4262432aab7_isrForCrosstalkSources_946479_51.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/2b19b237-a3a9-47ac-97c0-e4262432aab7_isrForCrosstalkSources_946479_51.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 8ff96c6b5af3359ea50f690b3086b7e0 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4faee463-19ef-4557-8797-e325a65b6567_isrForCrosstalkSources_946370_40.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4faee463-19ef-4557-8797-e325a65b6567_isrForCrosstalkSources_946370_40.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 581df1859afc9ae64237f09456f5b4a1 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/83e612a6-0868-436e-8cc9-fee9a3d89ca7_isrForCrosstalkSources_946479_55.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/83e612a6-0868-436e-8cc9-fee9a3d89ca7_isrForCrosstalkSources_946479_55.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 8138950c5e77bd0c765e9224bc284ae5 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/45b2b833-4f0f-4c26-8b78-4b47082f9efb_isrForCrosstalkSources_946371_27.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/45b2b833-4f0f-4c26-8b78-4b47082f9efb_isrForCrosstalkSources_946371_27.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash ff4e2affe66e306955f543b5a5d3a044 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/5401da2d-9f29-4a8a-88d5-28104bb18be1_isrForCrosstalkSources_946479_58.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/5401da2d-9f29-4a8a-88d5-28104bb18be1_isrForCrosstalkSources_946479_58.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 35bf92b26982bae6baa0e32f59236cef parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6cbd0cda-6fb5-4476-a576-b4860b0ecc80_isrForCrosstalkSources_946477_27.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6cbd0cda-6fb5-4476-a576-b4860b0ecc80_isrForCrosstalkSources_946477_27.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 0a2836b3c46cea33fdf74d3fe1d014f4 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/1f6c0c67-81f6-48cc-b9f4-86ea56a4c1e5_isrForCrosstalkSources_946478_50.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/1f6c0c67-81f6-48cc-b9f4-86ea56a4c1e5_isrForCrosstalkSources_946478_50.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash b5c59bced54b83c1fc348aa47b54a15f parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e3885461-e6b3-444e-86bd-a1444bbce5a7_isrForCrosstalkSources_946371_28.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e3885461-e6b3-444e-86bd-a1444bbce5a7_isrForCrosstalkSources_946371_28.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 158fd010a695770ad0c30cb2affce91f parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/f1e4df14-855f-4133-9baa-41d3a14ab688_isrForCrosstalkSources_946371_6.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/f1e4df14-855f-4133-9baa-41d3a14ab688_isrForCrosstalkSources_946371_6.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 51e74ec8ba6b5cbe47e6d28af0c93cec parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/30cdf018-0e5a-467c-b886-a9d40f121c7d_isrForCrosstalkSources_946369_52.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/30cdf018-0e5a-467c-b886-a9d40f121c7d_isrForCrosstalkSources_946369_52.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 8a1649b54f6fe0ff6281e6a9171fe658 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4702096b-ea52-45de-8cf3-006ffb04096e_isrForCrosstalkSources_946369_6.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4702096b-ea52-45de-8cf3-006ffb04096e_isrForCrosstalkSources_946369_6.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 5ea8a728860f2682a806303e3d0ccc7d parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9d871c65-78fe-40a2-9cb5-229d53daab40_isrForCrosstalkSources_946477_5.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9d871c65-78fe-40a2-9cb5-229d53daab40_isrForCrosstalkSources_946477_5.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash f21df2237811b99a6561e3e05efb975d parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/9293570c-aeb6-477d-aa7e-69837bea5c6d_isrForCrosstalkSources_946478_31.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/9293570c-aeb6-477d-aa7e-69837bea5c6d_isrForCrosstalkSources_946478_31.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 4cc0c6e47fab68cb384ea55b506438b0 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2d1af336-7710-4c5b-bd0f-e35a21523da3_isrForCrosstalkSources_946369_25.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2d1af336-7710-4c5b-bd0f-e35a21523da3_isrForCrosstalkSources_946369_25.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 798fbe063141c4c6cd1cc6fddcdf403f parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/02a74268-0989-4c9f-994e-8754a216c399_isrForCrosstalkSources_946370_19.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/02a74268-0989-4c9f-994e-8754a216c399_isrForCrosstalkSources_946370_19.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 31ace38ee05105f0239101a3702625cc parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/2c6ba1c2-fef4-4136-bc8d-7c83e25766e2_isrForCrosstalkSources_946478_55.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/2c6ba1c2-fef4-4136-bc8d-7c83e25766e2_isrForCrosstalkSources_946478_55.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 7b3f6df0a62d02f4116ee1471697644f parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/dd0cab6d-8b77-4825-b363-27f4cec77740_isrForCrosstalkSources_946371_3.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/dd0cab6d-8b77-4825-b363-27f4cec77740_isrForCrosstalkSources_946371_3.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 0a0056f14c4e17762523f1f28a33c973 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/140fd016-5c47-4166-97d2-40d8b9b6c56c_isrForCrosstalkSources_946479_6.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/140fd016-5c47-4166-97d2-40d8b9b6c56c_isrForCrosstalkSources_946479_6.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 747a6f393b9bb7f762ded338d1f88eee parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a240fcdb-d08e-40e3-a52f-763dcfde4164_isrForCrosstalkSources_946478_7.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a240fcdb-d08e-40e3-a52f-763dcfde4164_isrForCrosstalkSources_946478_7.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash cdb53b1499ba11e4845c1b736d87a26b parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/44bdc3a2-63dd-4355-9dd2-7dbc64cf95ff_isrForCrosstalkSources_946479_21.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/44bdc3a2-63dd-4355-9dd2-7dbc64cf95ff_isrForCrosstalkSources_946479_21.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 0671872d3e1341d56df16118098fd0cf parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6e2d3f32-f7cc-4192-897b-69ee0f272c13_isrForCrosstalkSources_946371_31.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6e2d3f32-f7cc-4192-897b-69ee0f272c13_isrForCrosstalkSources_946371_31.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 420a9ea859cf9f9cb2a24425a2a483ac parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/fd77d0b3-f598-4f2d-b694-a6487f578f1e_isrForCrosstalkSources_946478_40.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/fd77d0b3-f598-4f2d-b694-a6487f578f1e_isrForCrosstalkSources_946478_40.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash e42330b5fdeb8591392a84a27edf87c9 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6921b943-6ce9-43e9-ac9b-caf10068096a_isrForCrosstalkSources_946371_55.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6921b943-6ce9-43e9-ac9b-caf10068096a_isrForCrosstalkSources_946371_55.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash fb6f239e0902c388cf1f2a8ce7c5a677 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f98b2c2b-a12f-43aa-a652-7d62f27bab3e_isrForCrosstalkSources_946477_18.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f98b2c2b-a12f-43aa-a652-7d62f27bab3e_isrForCrosstalkSources_946477_18.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash f975bdac4cf697b15dcfd6cdaf5b2496 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d7cfeac9-4fc5-4a82-b915-22a96f5b348a_isrForCrosstalkSources_946371_44.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d7cfeac9-4fc5-4a82-b915-22a96f5b348a_isrForCrosstalkSources_946371_44.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash ddc0bcf7a2983204a374215592f7d5a2 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/48951240-979b-4d58-adde-22b069f1f603_isrForCrosstalkSources_946370_4.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/48951240-979b-4d58-adde-22b069f1f603_isrForCrosstalkSources_946370_4.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash d6e397cd9d399d45577310a14d807ea0 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6e17cf75-988e-4c72-98e7-d08c877db52b_isrForCrosstalkSources_946477_6.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6e17cf75-988e-4c72-98e7-d08c877db52b_isrForCrosstalkSources_946477_6.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 12b1dab222aac7ebe05de6cd4e8c5e42 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/5bde6491-37d1-46f6-9cd7-594c80723114_isrForCrosstalkSources_946477_58.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/5bde6491-37d1-46f6-9cd7-594c80723114_isrForCrosstalkSources_946477_58.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 76961ad00b65d981c56306ac0bc652e2 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1b68e782-fc79-43d6-acd8-319c5005da28_isrForCrosstalkSources_946369_58.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1b68e782-fc79-43d6-acd8-319c5005da28_isrForCrosstalkSources_946369_58.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash ef9c9a1b914cd0b8a6689dbbe3712f96 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/25e69276-bfd6-4bf8-a095-207e00809919_isrForCrosstalkSources_946479_44.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/25e69276-bfd6-4bf8-a095-207e00809919_isrForCrosstalkSources_946479_44.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash cd35a0a8741b82b4fc97fa0246770f36 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/dc3f0df5-edfc-4601-be7c-0d618c9670a4_isrForCrosstalkSources_946479_31.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/dc3f0df5-edfc-4601-be7c-0d618c9670a4_isrForCrosstalkSources_946479_31.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 2a4f1954e69a1b02377a7cded7f43fac parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d73ceabc-ae6c-4f7e-846e-33f1f061dcc1_isrForCrosstalkSources_946371_7.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d73ceabc-ae6c-4f7e-846e-33f1f061dcc1_isrForCrosstalkSources_946371_7.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash e8693f976257a5be257e611089eca811 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d3c63323-b9af-4edb-b88b-4e8f28f441db_isrForCrosstalkSources_946478_59.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d3c63323-b9af-4edb-b88b-4e8f28f441db_isrForCrosstalkSources_946478_59.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash f9a0b8d9269843602a610ce8d7339a57 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/93b8ff06-bcc3-461c-832f-dfe268aa17e7_isrForCrosstalkSources_946479_26.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/93b8ff06-bcc3-461c-832f-dfe268aa17e7_isrForCrosstalkSources_946479_26.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash afd8e5e033a6f0977f87ff304fbeb2ca parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/1980e801-91fd-4750-ba4b-0622b785c66b_isrForCrosstalkSources_946479_2.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/1980e801-91fd-4750-ba4b-0622b785c66b_isrForCrosstalkSources_946479_2.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 49fba05e92b764a2eeac6c290d4fbb89 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0da1a2a4-c1f0-4484-8442-dce2befdb4d0_isrForCrosstalkSources_946477_48.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0da1a2a4-c1f0-4484-8442-dce2befdb4d0_isrForCrosstalkSources_946477_48.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 83ad4449d698df076f853f84dca456e8 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e181bf53-fe38-402f-a243-adabba0db748_isrForCrosstalkSources_946478_35.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e181bf53-fe38-402f-a243-adabba0db748_isrForCrosstalkSources_946478_35.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash c023265a44238f3f9b7661120419c54c parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/a7193dc7-3a73-4d2a-83e3-9cf11117f78b_isrForCrosstalkSources_946370_20.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/a7193dc7-3a73-4d2a-83e3-9cf11117f78b_isrForCrosstalkSources_946370_20.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash a73d1caacdfdb0f8025e643c9a718a1f parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/630cee2a-48f4-41a7-bd82-7b2d4880a3b6_isrForCrosstalkSources_946479_14.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/630cee2a-48f4-41a7-bd82-7b2d4880a3b6_isrForCrosstalkSources_946479_14.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 170c89feb552bd6a956917fbd7995af8 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/498bffc5-976d-40a2-825a-b7b169e04bf0_isrForCrosstalkSources_946370_50.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/498bffc5-976d-40a2-825a-b7b169e04bf0_isrForCrosstalkSources_946370_50.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 606e488a3062f8bdab58214c73609677 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/9378bef4-6902-431f-a077-2b84da694ff8_isrForCrosstalkSources_946370_61.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/9378bef4-6902-431f-a077-2b84da694ff8_isrForCrosstalkSources_946370_61.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash e74d9ca40f036421ed538540f028fde9 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/d7f68d1b-a690-4a47-86a1-2a9610397ead_isrForCrosstalkSources_946370_48.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/d7f68d1b-a690-4a47-86a1-2a9610397ead_isrForCrosstalkSources_946370_48.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash d3be5390680ab975e7f2a1238dceafdb parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/15e59511-2ec8-47fd-a48d-52c3371f1138_isrForCrosstalkSources_946370_26.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/15e59511-2ec8-47fd-a48d-52c3371f1138_isrForCrosstalkSources_946370_26.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 6ac7923a382d808ae3e3cb970dba78a7 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4124479d-fd9e-4219-8339-15425ad5b04f_isrForCrosstalkSources_946369_50.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4124479d-fd9e-4219-8339-15425ad5b04f_isrForCrosstalkSources_946369_50.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash da55404abaa894f96d7e6f1339fc8f0b parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0ac4dd39-8e5f-44f0-8317-c33b5b2849a8_isrForCrosstalkSources_946370_57.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0ac4dd39-8e5f-44f0-8317-c33b5b2849a8_isrForCrosstalkSources_946370_57.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 1102e814215cb10d2a80e2cc1ccc755e parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c83fcc71-7982-4a43-a6c6-0f9ed6ad26e1_isrForCrosstalkSources_946479_25.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c83fcc71-7982-4a43-a6c6-0f9ed6ad26e1_isrForCrosstalkSources_946479_25.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 24050ed132b6a5f69148caea947dc81b parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d387e504-00fb-4d9a-8f55-8b75f30d1428_isrForCrosstalkSources_946371_15.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d387e504-00fb-4d9a-8f55-8b75f30d1428_isrForCrosstalkSources_946371_15.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 273de1c2214e0a81063ee8843156cc17 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/134c748c-e33b-4f93-8b81-407c8861cc92_isrForCrosstalkSources_946478_12.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/134c748c-e33b-4f93-8b81-407c8861cc92_isrForCrosstalkSources_946478_12.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash b2876f37011a4b40875eafee067e4753 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/50982bbe-5b4e-4269-8ffb-aa94cef7cde1_isrForCrosstalkSources_946479_45.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/50982bbe-5b4e-4269-8ffb-aa94cef7cde1_isrForCrosstalkSources_946479_45.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash aeb24e914e9d53dfb538ea007fa8b154 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6f746e01-a308-4dd6-bf6b-40cb7bd79a71_isrForCrosstalkSources_946371_59.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6f746e01-a308-4dd6-bf6b-40cb7bd79a71_isrForCrosstalkSources_946371_59.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash fe724735a722c5d797cb3e0fb7420e33 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/aea4fc84-f72a-40dd-9477-a5ce5b27f22b_isrForCrosstalkSources_946371_62.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/aea4fc84-f72a-40dd-9477-a5ce5b27f22b_isrForCrosstalkSources_946371_62.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash df20b6b11a33b6bf843a2e80230515cb parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0923c487-13d3-4649-a281-dd526341aef5_isrForCrosstalkSources_946477_33.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0923c487-13d3-4649-a281-dd526341aef5_isrForCrosstalkSources_946477_33.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash d6dce8f1e02c66e8f2acf38e98b3ff56 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9200e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c4086778-c8ac-420c-9f98-ae24054808bc_isrForCrosstalkSources_946369_32.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c4086778-c8ac-420c-9f98-ae24054808bc_isrForCrosstalkSources_946369_32.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash b05e38d5b959d1d9b7660eb26d3644c3 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5083b6f5-49dc-4315-af83-db9f567ce092_isrForCrosstalkSources_946369_44.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5083b6f5-49dc-4315-af83-db9f567ce092_isrForCrosstalkSources_946369_44.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 49f978599e483e36863da311b1f03fbb parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9202c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/581ad702-ad4e-4783-8aac-ef1322502524_isrForCrosstalkSources_946370_22.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/581ad702-ad4e-4783-8aac-ef1322502524_isrForCrosstalkSources_946370_22.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 880d308d885deb633478d46d3c9f3acb parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ca2d5076-0492-4050-a68d-e5c61d2356aa_isrForCrosstalkSources_946478_1.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ca2d5076-0492-4050-a68d-e5c61d2356aa_isrForCrosstalkSources_946478_1.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 43f1ac2f1994ea29976e8e00e70fc6c8 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/9f60ea8a-88e3-4212-ba00-2b13ed1400a7_isrForCrosstalkSources_946371_43.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/9f60ea8a-88e3-4212-ba00-2b13ed1400a7_isrForCrosstalkSources_946371_43.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash b480857b3b56ac9cfaa3d78bb8c389c7 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/02efcb4f-7142-4832-bd6d-7b304b4e4fce_isrForCrosstalkSources_946369_19.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/02efcb4f-7142-4832-bd6d-7b304b4e4fce_isrForCrosstalkSources_946369_19.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash ea3b6bc7adc300dc80f3643f41588bdf parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6fec7f49-e661-4a0a-ae9b-017f11b580f4_isrForCrosstalkSources_946478_44.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6fec7f49-e661-4a0a-ae9b-017f11b580f4_isrForCrosstalkSources_946478_44.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 1bd06f5b052a71f1f753b1813787f780 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9205e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/64dd8b81-158d-4972-8e44-38e4f74bedba_isrForCrosstalkSources_946370_52.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/64dd8b81-158d-4972-8e44-38e4f74bedba_isrForCrosstalkSources_946370_52.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash ac20067b62d6498947755ee40aef6d39 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3249015c-ec8b-42e4-874c-1ea3223c1a39_isrForCrosstalkSources_946479_61.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3249015c-ec8b-42e4-874c-1ea3223c1a39_isrForCrosstalkSources_946479_61.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 402bba84391b7645418fcec4538b3c6d parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6cc8da6c-d332-40b9-983e-bf3751f07b92_isrForCrosstalkSources_946478_37.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6cc8da6c-d332-40b9-983e-bf3751f07b92_isrForCrosstalkSources_946478_37.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 6ee4b063c0b6a05c553ad89e4ad99a23 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/463ce0ca-82ed-44eb-ad34-3bebac9a3213_isrForCrosstalkSources_946370_6.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/463ce0ca-82ed-44eb-ad34-3bebac9a3213_isrForCrosstalkSources_946370_6.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash cb8ec27756ea49088796e93e89df07dd parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9207c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a487eac3-a072-41c3-ab46-8d0777c4a33d_isrForCrosstalkSources_946478_54.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a487eac3-a072-41c3-ab46-8d0777c4a33d_isrForCrosstalkSources_946478_54.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 51a5fd7cf987ec3ab0f63fc8226149a1 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/7d37e929-aeed-4083-a3d9-7be660911548_isrForCrosstalkSources_946370_36.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/7d37e929-aeed-4083-a3d9-7be660911548_isrForCrosstalkSources_946370_36.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash ae37e3cea420b04e07abf4c020cd3855 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5c8a4f16-d11d-4c34-b253-f1282bce87ca_isrForCrosstalkSources_946369_22.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5c8a4f16-d11d-4c34-b253-f1282bce87ca_isrForCrosstalkSources_946369_22.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash ebf0a6e5466791d8bba0407ba0d98475 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9209a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/dbdfeca6-4c80-487a-a184-1c9fa7c2357d_isrForCrosstalkSources_946477_40.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/dbdfeca6-4c80-487a-a184-1c9fa7c2357d_isrForCrosstalkSources_946477_40.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 6c4b2e5db68250e41b6c79a1cb95a55d parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2bcd9380-9546-4335-b254-808f4ba547dc_isrForCrosstalkSources_946477_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2bcd9380-9546-4335-b254-808f4ba547dc_isrForCrosstalkSources_946477_2.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash a0dc71d2978d3b67cf3e50b2b0e3cd67 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d6f5e1f2-4c77-43d1-9209-2279395d4ae2_isrForCrosstalkSources_946478_16.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d6f5e1f2-4c77-43d1-9209-2279395d4ae2_isrForCrosstalkSources_946478_16.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 49f9c26405d3dfdc050b01eee2150618 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/bd5c48c2-e2d9-41f9-90ff-044752d7e9bd_isrForCrosstalkSources_946370_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/bd5c48c2-e2d9-41f9-90ff-044752d7e9bd_isrForCrosstalkSources_946370_2.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 0cdbad83151fa41cc048ede251a989f4 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/8e9ba0f5-a32e-457b-bfe6-f465a9cd5dbb_isrForCrosstalkSources_946477_28.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/8e9ba0f5-a32e-457b-bfe6-f465a9cd5dbb_isrForCrosstalkSources_946477_28.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 5f80452cc60aa9a7bd1b625191b791ae parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d7aae3c7-d542-46c7-9a9c-3badb320934d_isrForCrosstalkSources_946478_13.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d7aae3c7-d542-46c7-9a9c-3badb320934d_isrForCrosstalkSources_946478_13.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 820042b7cb7487c07df30aa400cd4071 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/e938c4de-8308-40f2-a9a4-7833d5d451f3_isrForCrosstalkSources_946369_40.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/e938c4de-8308-40f2-a9a4-7833d5d451f3_isrForCrosstalkSources_946369_40.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 4f26caf46ad030e1e1444fe8f64ff2f3 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/4e40be26-0305-4e78-89d8-6bba1f5d4df1_isrForCrosstalkSources_946479_37.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/4e40be26-0305-4e78-89d8-6bba1f5d4df1_isrForCrosstalkSources_946479_37.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 5efa6e49e064f6b74a2f95193e17ef17 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/d10daa43-88c4-464d-b0e9-20b63ea1072f_isrForCrosstalkSources_946477_32.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/d10daa43-88c4-464d-b0e9-20b63ea1072f_isrForCrosstalkSources_946477_32.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash ca04db9837f7fc108600e7b499fc358e parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e445590e-cbb9-4c35-95d5-3e408106efee_isrForCrosstalkSources_946479_40.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e445590e-cbb9-4c35-95d5-3e408106efee_isrForCrosstalkSources_946479_40.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 72b755c148a429bfcf5b2bab1befaa82 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b920fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/f7209d4a-ac40-4a06-b990-702a9d4c616b_isrForCrosstalkSources_946369_21.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/f7209d4a-ac40-4a06-b990-702a9d4c616b_isrForCrosstalkSources_946369_21.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash d4adff40ec398dd426cd45d807d4f4fa parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/99d3d3be-2769-4db4-9a24-f007d7c0041d_isrForCrosstalkSources_946369_62.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/99d3d3be-2769-4db4-9a24-f007d7c0041d_isrForCrosstalkSources_946369_62.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 39be7dbd947b3371222e4c9d987e7cdd parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3f2bf86e-0ab7-443c-bd98-9882573497f8_isrForCrosstalkSources_946477_20.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3f2bf86e-0ab7-443c-bd98-9882573497f8_isrForCrosstalkSources_946477_20.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 2839f664bf11034777bb4f01bbbc42f2 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1858c8c9-a413-459e-b019-38ef2969dcf9_isrForCrosstalkSources_946370_25.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1858c8c9-a413-459e-b019-38ef2969dcf9_isrForCrosstalkSources_946370_25.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash d29b5220548d310d60e941b3bca671e5 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/93de60b2-ff80-4b9f-83fd-1e6f95fc94d4_isrForCrosstalkSources_946479_30.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/93de60b2-ff80-4b9f-83fd-1e6f95fc94d4_isrForCrosstalkSources_946479_30.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 175b85b368b661fd782ab401f0311ac3 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ca5eb188-af0b-47de-b9d0-8a714968aacd_isrForCrosstalkSources_946478_6.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ca5eb188-af0b-47de-b9d0-8a714968aacd_isrForCrosstalkSources_946478_6.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 957808e4c91d18dfb2752e5a129cbe35 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/3c39584d-9233-4577-a582-e42061b33633_isrForCrosstalkSources_946370_46.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/3c39584d-9233-4577-a582-e42061b33633_isrForCrosstalkSources_946370_46.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash fdbe0fc7f3dbff54d9246e5ce5b406fb parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f8479c10-7c84-41d9-a2dd-ef8b145f72f9_isrForCrosstalkSources_946478_36.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f8479c10-7c84-41d9-a2dd-ef8b145f72f9_isrForCrosstalkSources_946478_36.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 7e3acfab16766d0458003eed92d3e943 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/500c8c3a-215e-41cb-a5b1-af6134d3c393_isrForCrosstalkSources_946479_23.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/500c8c3a-215e-41cb-a5b1-af6134d3c393_isrForCrosstalkSources_946479_23.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 8f20c8c917b399ee79d5df6d7152a3ed parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9214e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0ec2425f-a5ee-429e-83cb-c36a4325bd0e_isrForCrosstalkSources_946371_16.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0ec2425f-a5ee-429e-83cb-c36a4325bd0e_isrForCrosstalkSources_946371_16.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 9ac0322d9bd610d63b082f6a5de9a768 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c6d6a430-4273-4c75-b1d0-a18349b1de2e_isrForCrosstalkSources_946478_10.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c6d6a430-4273-4c75-b1d0-a18349b1de2e_isrForCrosstalkSources_946478_10.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 6f46fea408a5f10506f302380d5e30d1 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9216c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/fd77f74c-1ac5-4d52-8782-631eb25cff46_isrForCrosstalkSources_946369_30.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/fd77f74c-1ac5-4d52-8782-631eb25cff46_isrForCrosstalkSources_946369_30.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash cf69832902c374dd0660ce00cc492fcf parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/1839b9b2-ae84-4bb5-b9b7-2e96fae4ec66_isrForCrosstalkSources_946479_3.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/1839b9b2-ae84-4bb5-b9b7-2e96fae4ec66_isrForCrosstalkSources_946479_3.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 1d5d0d0aacdcf861f5c3e85f4ced51bc parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9218a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/ea84f01a-995a-4d75-93a1-d652b8948638_isrForCrosstalkSources_946369_31.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/ea84f01a-995a-4d75-93a1-d652b8948638_isrForCrosstalkSources_946369_31.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash ac3758e8454d1466aa2e1c181a617d95 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6ad230a0-f9fb-4abe-aeac-6cbee9cdd410_isrForCrosstalkSources_946478_28.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6ad230a0-f9fb-4abe-aeac-6cbee9cdd410_isrForCrosstalkSources_946478_28.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 413e9c4f3ba0cce9250683f53d168f7d parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/89ec25c0-27d2-4210-8259-83f06774117c_isrForCrosstalkSources_946478_26.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/89ec25c0-27d2-4210-8259-83f06774117c_isrForCrosstalkSources_946478_26.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 496d409509b5438cfac0a67625e7bf62 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9219e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/fbe0627e-3298-41ce-9299-07eb41843371_isrForCrosstalkSources_946477_47.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/fbe0627e-3298-41ce-9299-07eb41843371_isrForCrosstalkSources_946477_47.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash dee3371fdfd0e37e225700b632fe9ae3 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/0e1a4e91-b3c0-4e2b-b892-d6ae189ca0c8_isrForCrosstalkSources_946479_57.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/0e1a4e91-b3c0-4e2b-b892-d6ae189ca0c8_isrForCrosstalkSources_946479_57.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 0b49af9e1b14894f81da8eb464069873 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/62b2379d-c56f-443f-9e58-e1b548864015_isrForCrosstalkSources_946369_45.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/62b2379d-c56f-443f-9e58-e1b548864015_isrForCrosstalkSources_946369_45.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 83703b286350bb472b5fe52715afe58f parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/84e890e2-06fb-4c33-9b96-c3d232438871_isrForCrosstalkSources_946370_14.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/84e890e2-06fb-4c33-9b96-c3d232438871_isrForCrosstalkSources_946370_14.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash dc1216159a55500b0bb83add19c359a2 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e33397d8-883d-40d2-87f9-560aa987a36b_isrForCrosstalkSources_946479_56.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e33397d8-883d-40d2-87f9-560aa987a36b_isrForCrosstalkSources_946479_56.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 36bc67a6033c5cb5643155cd006e5acd parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/988d9c5c-0270-49c2-a16f-230608bf1405_isrForCrosstalkSources_946371_40.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/988d9c5c-0270-49c2-a16f-230608bf1405_isrForCrosstalkSources_946371_40.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash daa6a13e6cdb4c21014930f26f14f9eb parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/8edb4d46-77fe-4847-a7a9-a97be9875fe8_isrForCrosstalkSources_946370_8.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/8edb4d46-77fe-4847-a7a9-a97be9875fe8_isrForCrosstalkSources_946370_8.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 8d4bd597705d2ff2b796f10ad3bcb703 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/826f55fd-13ba-48f6-a54a-c07c76622a75_isrForCrosstalkSources_946478_23.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/826f55fd-13ba-48f6-a54a-c07c76622a75_isrForCrosstalkSources_946478_23.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash f9eb2db7c8e8aa019504ea959d1e31d0 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/3ef9b635-c9f2-460d-827c-36d623edf9d1_isrForCrosstalkSources_946371_36.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/3ef9b635-c9f2-460d-827c-36d623edf9d1_isrForCrosstalkSources_946371_36.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 4a2c434efada220b5c1e3ca8e0cda509 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b921f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/a5515d24-605f-4349-9afc-0d8d75c757d7_isrForCrosstalkSources_946369_24.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/a5515d24-605f-4349-9afc-0d8d75c757d7_isrForCrosstalkSources_946369_24.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash fafb746b7a7f973ca36afd4ceb3c451b parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/2dc73e27-8136-42af-810b-d3e5390eb4c7_isrForCrosstalkSources_946479_39.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/2dc73e27-8136-42af-810b-d3e5390eb4c7_isrForCrosstalkSources_946479_39.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 894571f43fa42ea11eff71ce4aa6a312 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/c32fbb8b-fce2-4b90-8134-29fe27fe8d9b_isrForCrosstalkSources_946477_62.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/c32fbb8b-fce2-4b90-8134-29fe27fe8d9b_isrForCrosstalkSources_946477_62.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 384ca7d7b0c8d4ad9fcd0b63d5ee42a8 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/643875a3-e0fb-4c1d-9bc7-8d16ce864155_isrForCrosstalkSources_946477_41.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/643875a3-e0fb-4c1d-9bc7-8d16ce864155_isrForCrosstalkSources_946477_41.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash ef8c598847550b8cd0356b95e49e5242 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/f9fb522f-4977-40fd-a692-d51c9c851232_isrForCrosstalkSources_946370_3.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/f9fb522f-4977-40fd-a692-d51c9c851232_isrForCrosstalkSources_946370_3.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 8e179ea2f4c0c3beac4bedd18165ec6a parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/8d8a8c58-d2c9-4a94-9ec7-40dee328b5a2_isrForCrosstalkSources_946479_43.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/8d8a8c58-d2c9-4a94-9ec7-40dee328b5a2_isrForCrosstalkSources_946479_43.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 67d970b7855b6dc66c1d68701badf1b1 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a0a530f3-56ed-433d-bdd3-1cc62357685f_isrForCrosstalkSources_946478_41.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a0a530f3-56ed-433d-bdd3-1cc62357685f_isrForCrosstalkSources_946478_41.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 6acda13309f184fb68a206c3c43f5438 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9223e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4681e298-f395-466a-8af1-d87584e32717_isrForCrosstalkSources_946371_10.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4681e298-f395-466a-8af1-d87584e32717_isrForCrosstalkSources_946371_10.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 4b291246eba27aeef0373a03484a069a parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2f248384-9980-4cad-9e82-79c5bcbf950a_isrForCrosstalkSources_946369_37.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2f248384-9980-4cad-9e82-79c5bcbf950a_isrForCrosstalkSources_946369_37.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 577e5a11ad385f6d5cd1e6b10d7db8b5 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e5adc0e7-8fd5-4a6c-8aba-4454d426dee6_isrForCrosstalkSources_946477_9.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e5adc0e7-8fd5-4a6c-8aba-4454d426dee6_isrForCrosstalkSources_946477_9.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash cb3c2246c492aa85ba435fbdaea94d37 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9225c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0bc9f122-5922-44f9-bcaa-51c5ad62d2b4_isrForCrosstalkSources_946371_56.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0bc9f122-5922-44f9-bcaa-51c5ad62d2b4_isrForCrosstalkSources_946371_56.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 47f98d7bbf38e02b643d446f2b3278f4 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f309b70b-f7e9-4d5f-b722-aa4abba5e74b_isrForCrosstalkSources_946478_51.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f309b70b-f7e9-4d5f-b722-aa4abba5e74b_isrForCrosstalkSources_946478_51.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 790f03a14d5e0278794c452d039d32a3 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/861b1a92-9631-4d32-8874-106401a8c0d4_isrForCrosstalkSources_946478_21.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/861b1a92-9631-4d32-8874-106401a8c0d4_isrForCrosstalkSources_946478_21.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash fba5923148bcff2f276ea46f177e2003 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/10910ece-f063-4074-b65d-839112b0e6e0_isrForCrosstalkSources_946369_39.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/10910ece-f063-4074-b65d-839112b0e6e0_isrForCrosstalkSources_946369_39.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 5a062f3537dc3d34c8eb54033bdf0f95 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9227a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/813a6654-cb8b-41b3-9d45-d96363f1ccd8_isrForCrosstalkSources_946369_34.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/813a6654-cb8b-41b3-9d45-d96363f1ccd8_isrForCrosstalkSources_946369_34.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 032d86527086254bd86280f1979fe720 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9228e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3a0c5383-af37-45c2-b336-88edca89a472_isrForCrosstalkSources_946479_19.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3a0c5383-af37-45c2-b336-88edca89a472_isrForCrosstalkSources_946479_19.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 8a24c9c38cde060f11fb9e4f2878ca54 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0bb6ff00-a6b3-4e3b-af9a-992c791138b9_isrForCrosstalkSources_946369_7.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0bb6ff00-a6b3-4e3b-af9a-992c791138b9_isrForCrosstalkSources_946369_7.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash eaa7f082e8b3b92f04e31b318e323af9 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/dd9b1132-70cc-4ef3-8bec-5cf0b64a4ac8_isrForCrosstalkSources_946478_56.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/dd9b1132-70cc-4ef3-8bec-5cf0b64a4ac8_isrForCrosstalkSources_946478_56.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash f6abcf071d76ecc717182247ff4e871e parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/98afbdc1-e1db-465e-89aa-254758112f83_isrForCrosstalkSources_946370_23.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/98afbdc1-e1db-465e-89aa-254758112f83_isrForCrosstalkSources_946370_23.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 42c2169671b3c4c695c002fae1aca791 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/6df1d4f3-c789-441a-b8f5-fc0f316651b3_isrForCrosstalkSources_946369_1.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/6df1d4f3-c789-441a-b8f5-fc0f316651b3_isrForCrosstalkSources_946369_1.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 1bb8db2f3d7166c19c8bcd10dca10f00 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4442901d-17ac-427a-95af-88ada76f36c2_isrForCrosstalkSources_946477_15.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4442901d-17ac-427a-95af-88ada76f36c2_isrForCrosstalkSources_946477_15.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 4666fa49c68d898917f244612bf83948 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c1ab57d2-6192-48f6-a4e0-035da39292c3_isrForCrosstalkSources_946369_46.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c1ab57d2-6192-48f6-a4e0-035da39292c3_isrForCrosstalkSources_946369_46.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 5a24f636e02aa931544a426a7a29f28e parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3770f4bd-8827-4106-b70f-821a541f7264_isrForCrosstalkSources_946479_49.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3770f4bd-8827-4106-b70f-821a541f7264_isrForCrosstalkSources_946479_49.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash e5b8cd0d0c12418b9147f6717f604875 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0de15106-b63d-450c-a4d0-d6272104c24c_isrForCrosstalkSources_946369_4.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0de15106-b63d-450c-a4d0-d6272104c24c_isrForCrosstalkSources_946369_4.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash a49f43352b47c3e1e8ca5fe7d4577937 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/864d16b7-d0a9-48bf-89fa-8ae86edabc9b_isrForCrosstalkSources_946477_38.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/864d16b7-d0a9-48bf-89fa-8ae86edabc9b_isrForCrosstalkSources_946477_38.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 2aaef821033b6a030a208cdc83b31154 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/effcce2b-94d1-4590-980e-0579221185fe_isrForCrosstalkSources_946371_29.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/effcce2b-94d1-4590-980e-0579221185fe_isrForCrosstalkSources_946371_29.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 57f2b55863a22c0e78d29682c9b8a20c parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b922fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0974e617-a519-4278-8b1b-5bbd05b90efc_isrForCrosstalkSources_946370_31.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0974e617-a519-4278-8b1b-5bbd05b90efc_isrForCrosstalkSources_946370_31.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 755f7b1cd27454050995c7f0b00d8475 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/500c259e-2506-4bad-b0b0-9bdfd403cd42_isrForCrosstalkSources_946371_33.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/500c259e-2506-4bad-b0b0-9bdfd403cd42_isrForCrosstalkSources_946371_33.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash dddf8482ec032cd967049c21c069a82f parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/89ddeb0d-aabf-4e6b-81c3-18f2d95ef747_isrForCrosstalkSources_946478_33.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/89ddeb0d-aabf-4e6b-81c3-18f2d95ef747_isrForCrosstalkSources_946478_33.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 0e709484a362bbdaf0fe3ff438a6f20b parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4722ebf1-fea5-46e5-a865-102bec81638f_isrForCrosstalkSources_946371_51.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4722ebf1-fea5-46e5-a865-102bec81638f_isrForCrosstalkSources_946371_51.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash f6ae5e84e4a7aca1c05d32a7b173ee62 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/799d634c-5afe-4c6d-94ab-d3fa65bedf48_isrForCrosstalkSources_946371_53.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/799d634c-5afe-4c6d-94ab-d3fa65bedf48_isrForCrosstalkSources_946371_53.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 086e764aee346c14512ec50d845c8d8a parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9232e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/5f3d473e-6e78-4900-baba-e592277452be_isrForCrosstalkSources_946479_54.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/5f3d473e-6e78-4900-baba-e592277452be_isrForCrosstalkSources_946479_54.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 543e301ade364e67df0b46954edcbabe parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 23:13:07 proc_lsst.multi:146 [INFO] found job 20175 in provider local proc_lsst.multi INFO: found job 20175 in provider local parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/16acc3fb-6757-4f9f-b3a5-b62aa6204dd8_isrForCrosstalkSources_946369_11.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/16acc3fb-6757-4f9f-b3a5-b62aa6204dd8_isrForCrosstalkSources_946369_11.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash cd5b4fcd972d7da8424075bce81e24f0 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b8e3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e56103f2-84a1-4f72-ac6b-28886b5c171b_isrForCrosstalkSources_946478_46.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e56103f2-84a1-4f72-ac6b-28886b5c171b_isrForCrosstalkSources_946478_46.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 0f3e1f85c75827253d30f55135919dc3 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/930f0dcb-04ac-424b-b7ef-11343d9b5068_isrForCrosstalkSources_946370_7.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/930f0dcb-04ac-424b-b7ef-11343d9b5068_isrForCrosstalkSources_946370_7.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash b4a5eb5bad5ccd94261df586821cf17a parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9234c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2efbc69a-2f55-40a2-9f13-f44ad31badcb_isrForCrosstalkSources_946477_16.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2efbc69a-2f55-40a2-9f13-f44ad31badcb_isrForCrosstalkSources_946477_16.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash ae288a57a036fcb16b79455627cfad2e parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9236a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/df5a4d78-a8b7-4a2f-ab74-1a5575f4a135_isrForCrosstalkSources_946478_15.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/df5a4d78-a8b7-4a2f-ab74-1a5575f4a135_isrForCrosstalkSources_946478_15.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 6257e4f083310a0ebcc1394c2c2193c2 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0b588a42-f22d-42e9-a2a7-4ca0a28163aa_isrForCrosstalkSources_946477_42.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0b588a42-f22d-42e9-a2a7-4ca0a28163aa_isrForCrosstalkSources_946477_42.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash b8594082aac58f265735688b37fcffab parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/916f6a15-2932-4ed0-aa6a-6a19034c4451_isrForCrosstalkSources_946371_25.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/916f6a15-2932-4ed0-aa6a-6a19034c4451_isrForCrosstalkSources_946371_25.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 6c69892a95914708c03b476e8ba6466d parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9237e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6bf367fb-8022-4d9a-9408-2b41bc75de0f_isrForCrosstalkSources_946478_8.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6bf367fb-8022-4d9a-9408-2b41bc75de0f_isrForCrosstalkSources_946478_8.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 27da058f123b21213b8ee47f4214f2f5 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b9239c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 150 parsl.dataflow.strategy DEBUG: Executor multi has 150 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 37 more blocks parsl.executors.status_handling INFO: Scaling out by 37 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-13 23:13:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-13 23:13:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 23:13:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/5462ca9c-5499-4849-96dd-a0da2da2f0a1_isrForCrosstalkSources_946371_23.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/5462ca9c-5499-4849-96dd-a0da2da2f0a1_isrForCrosstalkSources_946371_23.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 2adb9f2308fe0e729194f64ee262c417 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9d0fa824-1bd1-48c7-bb59-cb557a92fca4_isrForCrosstalkSources_946479_36.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9d0fa824-1bd1-48c7-bb59-cb557a92fca4_isrForCrosstalkSources_946479_36.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 0606e30db141bc970f796fd0d6a55a28 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/886983e1-fbac-441e-a14e-b161b93e61f7_isrForCrosstalkSources_946479_48.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/886983e1-fbac-441e-a14e-b161b93e61f7_isrForCrosstalkSources_946479_48.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 97a51c5d759b59fe97274de9ee9f32f1 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/437c6410-6157-4ed6-8138-11fb1f5bbc4c_isrForCrosstalkSources_946478_27.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/437c6410-6157-4ed6-8138-11fb1f5bbc4c_isrForCrosstalkSources_946478_27.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 98c5e79ba7da44f51719959ba0286dd2 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0b32cca8-aa31-4f6e-937f-6fb76e202799_isrForCrosstalkSources_946370_17.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0b32cca8-aa31-4f6e-937f-6fb76e202799_isrForCrosstalkSources_946370_17.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 5b3f933242552c9ac4e3c951eca496eb parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9cafbe2b-7970-4363-bdf0-fd940f880fd4_isrForCrosstalkSources_946477_19.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9cafbe2b-7970-4363-bdf0-fd940f880fd4_isrForCrosstalkSources_946477_19.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash bb1adca75f49450014ebd1693e059c80 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f5b877fe-653a-4359-a0e8-38d7462ef016_isrForCrosstalkSources_946478_62.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f5b877fe-653a-4359-a0e8-38d7462ef016_isrForCrosstalkSources_946478_62.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash f7109631f4b0288684175e8cab56789c parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/54c3998c-d33a-4060-87f5-764117b0f80a_isrForCrosstalkSources_946479_11.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/54c3998c-d33a-4060-87f5-764117b0f80a_isrForCrosstalkSources_946479_11.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 13ec0cc5d262132b2b5bfb67a290d679 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/c3711d70-8b13-4a97-85ae-4e87b18c1637_isrForCrosstalkSources_946371_60.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/c3711d70-8b13-4a97-85ae-4e87b18c1637_isrForCrosstalkSources_946371_60.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 9c61160f6bf050d9471eb5d1ff529453 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/37c2d025-6933-49dd-b738-9e4311eb3c32_isrForCrosstalkSources_946371_41.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/37c2d025-6933-49dd-b738-9e4311eb3c32_isrForCrosstalkSources_946371_41.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 9e108354e89720864c0a56d006ca5d25 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3480e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4b9e7deb-3ebd-4efe-a61a-33e24b20d326_isrForCrosstalkSources_946371_50.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4b9e7deb-3ebd-4efe-a61a-33e24b20d326_isrForCrosstalkSources_946371_50.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash eb6ecae4e68437dabd19a4cc875d39c2 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a2aee55e-433f-48a6-85b8-e4a4d9603ddb_isrForCrosstalkSources_946371_8.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a2aee55e-433f-48a6-85b8-e4a4d9603ddb_isrForCrosstalkSources_946371_8.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 5a5fdfc4a0e01258fe945265fdb1285d parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/827f0d3c-fbea-4d74-a7fe-915c23fc6f79_isrForCrosstalkSources_946479_62.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/827f0d3c-fbea-4d74-a7fe-915c23fc6f79_isrForCrosstalkSources_946479_62.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 0a1f1d8efeaf159ea1bf6f403a3f6676 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/34f7c2ed-1712-41e2-bb98-c3aa3862cc1d_isrForCrosstalkSources_946371_14.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/34f7c2ed-1712-41e2-bb98-c3aa3862cc1d_isrForCrosstalkSources_946371_14.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 29f3ef56ecaa10ce2dace6ecfa388ef8 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3482c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/31e7b1c9-aff8-4da9-a13b-424ca6e3ade9_isrForCrosstalkSources_946478_49.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/31e7b1c9-aff8-4da9-a13b-424ca6e3ade9_isrForCrosstalkSources_946478_49.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 10470616df40f84263fe8597be53d9e5 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/389ca738-025b-47fd-a66c-d58dddd25b48_isrForCrosstalkSources_946479_34.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/389ca738-025b-47fd-a66c-d58dddd25b48_isrForCrosstalkSources_946479_34.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash edbdae557c82745d4ddbf97186667235 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c8115f97-e597-4ea9-a160-3a9e4436a5c5_isrForCrosstalkSources_946370_47.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c8115f97-e597-4ea9-a160-3a9e4436a5c5_isrForCrosstalkSources_946370_47.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 96d03ee87fd7ae750a80cff61585732e parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/8d10fa7b-32a9-49c7-83d2-c35b1fc3eced_isrForCrosstalkSources_946478_60.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/8d10fa7b-32a9-49c7-83d2-c35b1fc3eced_isrForCrosstalkSources_946478_60.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 0c7d02621c88b17307780cd65e5fa929 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4f6dacf5-c988-46bc-bbc2-b60df6b47576_isrForCrosstalkSources_946369_53.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4f6dacf5-c988-46bc-bbc2-b60df6b47576_isrForCrosstalkSources_946369_53.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 7f2a8ca5180c396befe7dd933b1b00b9 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3485e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/88417e92-eedc-4da4-9b6d-0c5edf534e29_isrForCrosstalkSources_946370_51.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/88417e92-eedc-4da4-9b6d-0c5edf534e29_isrForCrosstalkSources_946370_51.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 160fad007f2486cf1fe59b3c72436fcb parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4c1054e8-9c75-4a03-a672-a8df3f33fb42_isrForCrosstalkSources_946369_56.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4c1054e8-9c75-4a03-a672-a8df3f33fb42_isrForCrosstalkSources_946369_56.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash aceef3f578691f6f7565f9cc738b2e07 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e1c91ed3-28b2-471d-957d-d6e8a1fd5a54_isrForCrosstalkSources_946479_5.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e1c91ed3-28b2-471d-957d-d6e8a1fd5a54_isrForCrosstalkSources_946479_5.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 1af3dc83cbdaf9b400e36bb5cf097d70 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4a4eb0c0-87ab-4893-8525-f97474a5ced6_isrForCrosstalkSources_946477_21.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4a4eb0c0-87ab-4893-8525-f97474a5ced6_isrForCrosstalkSources_946477_21.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 7af6a6b22e0831da87db9c33200a0125 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3487c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/510c5daf-5017-4791-9847-3ace8ee1eee7_isrForCrosstalkSources_946479_7.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/510c5daf-5017-4791-9847-3ace8ee1eee7_isrForCrosstalkSources_946479_7.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 4ddd7cdccda6cdf799fcc649cb272f8b parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3489a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/5a1bebd6-4ff8-4609-adae-1c78e6254b27_isrForCrosstalkSources_946370_34.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/5a1bebd6-4ff8-4609-adae-1c78e6254b27_isrForCrosstalkSources_946370_34.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash ccecf045f1b826d9c5a0471fd4f6a51b parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c00ccc8f-2079-4000-bc07-6c05e1e5435d_isrForCrosstalkSources_946479_27.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c00ccc8f-2079-4000-bc07-6c05e1e5435d_isrForCrosstalkSources_946479_27.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash dbe72497e98b9debeea7e3394cc012e6 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/846f330a-0084-4429-8a27-b36eec8ea226_isrForCrosstalkSources_946477_30.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/846f330a-0084-4429-8a27-b36eec8ea226_isrForCrosstalkSources_946477_30.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash f895f1e5bd210896a3c92446293bb160 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/ec9ce60b-1c62-446a-a6b4-762cf601900a_isrForCrosstalkSources_946479_16.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/ec9ce60b-1c62-446a-a6b4-762cf601900a_isrForCrosstalkSources_946479_16.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 582ef90c00e298c670dd8e7a187f1d14 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 23:13:07 proc_lsst.multi:170 [INFO] [multi] job_id 17025207 proc_lsst.multi INFO: [multi] job_id 17025207 2024-03-13 23:13:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025207 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025207 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17025207 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-13 23:13:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-13 23:13:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1490802c-7152-4ca8-86ba-9f9a96de165b_isrForCrosstalkSources_946369_61.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1490802c-7152-4ca8-86ba-9f9a96de165b_isrForCrosstalkSources_946369_61.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash a9738479699053f4c4d3095968a95b96 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e4dc26f1-2f36-47ce-bebf-199e32475cfb_isrForCrosstalkSources_946478_18.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e4dc26f1-2f36-47ce-bebf-199e32475cfb_isrForCrosstalkSources_946478_18.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 7e933fd874f27725ea3fe6f294528bcc parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/41b64316-42c3-4f7c-8d2b-c17be6f407b6_isrForCrosstalkSources_946478_3.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/41b64316-42c3-4f7c-8d2b-c17be6f407b6_isrForCrosstalkSources_946478_3.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 97e1f806d04461631bf31571fd63a5d7 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42b923d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/67a1605f-a864-4e06-884e-bab1a89c6368_isrForCrosstalkSources_946371_20.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/67a1605f-a864-4e06-884e-bab1a89c6368_isrForCrosstalkSources_946371_20.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash bb7e1cff49da4f4bba2f3f246d842c21 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f41b50ee-0563-4574-b638-0226e94267e0_isrForCrosstalkSources_946477_25.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f41b50ee-0563-4574-b638-0226e94267e0_isrForCrosstalkSources_946477_25.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 90f2432ff8a37332848c3e6faaf5e0a4 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/26460932-01be-439e-bb57-623c7a5806c7_isrForCrosstalkSources_946369_13.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/26460932-01be-439e-bb57-623c7a5806c7_isrForCrosstalkSources_946369_13.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 2d5e598b352bb54cafef436a5263fa98 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0eb25c5d-8ee0-47e0-ad45-db504847bcc7_isrForCrosstalkSources_946477_35.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0eb25c5d-8ee0-47e0-ad45-db504847bcc7_isrForCrosstalkSources_946477_35.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 66efcc3977d2658b9b8e31f7ca25ebba parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/6ea2e8e7-4909-4b0d-a665-20b383962b48_isrForCrosstalkSources_946369_28.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/6ea2e8e7-4909-4b0d-a665-20b383962b48_isrForCrosstalkSources_946369_28.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 7f7c4bf23ed0d3c90380e75fc1c4ef83 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/f1027f15-facc-4ea9-ac62-e4a224a18e50_isrForCrosstalkSources_946371_11.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/f1027f15-facc-4ea9-ac62-e4a224a18e50_isrForCrosstalkSources_946371_11.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 74abcdab785c8f67799b021aa900b44a parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/bcf0f4dc-4276-47ab-9ca8-d290748c2e4f_isrForCrosstalkSources_946370_5.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/bcf0f4dc-4276-47ab-9ca8-d290748c2e4f_isrForCrosstalkSources_946370_5.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 71e6118c76d178a6b7756f0d8b1f259a parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/fe5d1ca7-ae1c-429a-8501-38ef60c576de_isrForCrosstalkSources_946478_42.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/fe5d1ca7-ae1c-429a-8501-38ef60c576de_isrForCrosstalkSources_946478_42.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash d22a94c96e8eea542802302929aa5b80 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/29ed89d4-6b6d-4639-944f-fa4da6927220_isrForCrosstalkSources_946371_32.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/29ed89d4-6b6d-4639-944f-fa4da6927220_isrForCrosstalkSources_946371_32.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 7112fe3a6830e6e2ef689e893991e435 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/9853c604-9041-474c-8ca2-f890aff52176_isrForCrosstalkSources_946370_27.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/9853c604-9041-474c-8ca2-f890aff52176_isrForCrosstalkSources_946370_27.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 81c1c68ad3b5f73034e65c0232de366e parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4a3e51fc-89b9-4118-a59a-5dfdd5e9b32c_isrForCrosstalkSources_946477_23.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4a3e51fc-89b9-4118-a59a-5dfdd5e9b32c_isrForCrosstalkSources_946477_23.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash fd4aca244edae083a9e647f8052e4abe parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/df8a6635-50a8-4f9e-b0f9-49a8636f26d7_isrForCrosstalkSources_946478_5.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/df8a6635-50a8-4f9e-b0f9-49a8636f26d7_isrForCrosstalkSources_946478_5.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash da565a8900725dab5b9d40f244e23595 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/38f3a9fe-d60b-4e8e-94ac-5d7d3a0fd784_isrForCrosstalkSources_946369_57.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/38f3a9fe-d60b-4e8e-94ac-5d7d3a0fd784_isrForCrosstalkSources_946369_57.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 060aad550375a21e956a590b69a1a07d parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/8aec4174-a82a-44be-a73a-9d4cc2e3e5f3_isrForCrosstalkSources_946371_38.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/8aec4174-a82a-44be-a73a-9d4cc2e3e5f3_isrForCrosstalkSources_946371_38.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash b4186636829cef3b437d2dac4871f412 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1129e395-a0fb-458e-9ac3-4c0f2ae61792_isrForCrosstalkSources_946370_55.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1129e395-a0fb-458e-9ac3-4c0f2ae61792_isrForCrosstalkSources_946370_55.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 95aefdc2108c7525a34690ccf8fe4909 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3493a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/f50bf6a3-73fd-42ff-97e2-304abc5abbd3_isrForCrosstalkSources_946369_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/f50bf6a3-73fd-42ff-97e2-304abc5abbd3_isrForCrosstalkSources_946369_2.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 987dbaf40c1c61a98352715454b12f27 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e6eff6f4-3807-411b-8dcc-5a3fc4f7cafa_isrForCrosstalkSources_946477_52.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e6eff6f4-3807-411b-8dcc-5a3fc4f7cafa_isrForCrosstalkSources_946477_52.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 8edfef2b9acdd7f4bdd188bd96d84ce0 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/7fb7ad48-921a-4e9a-800d-52b05ab82f1e_isrForCrosstalkSources_946477_14.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/7fb7ad48-921a-4e9a-800d-52b05ab82f1e_isrForCrosstalkSources_946477_14.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 60fe37d9b0223dc5757736da52caf965 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3494e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e3b47d42-45c3-474e-b406-cc8df6b02168_isrForCrosstalkSources_946370_39.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e3b47d42-45c3-474e-b406-cc8df6b02168_isrForCrosstalkSources_946370_39.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash f61968eb8e07f72466e45336ef278ab3 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/52b2f4f3-66cb-4317-9323-e15a1ce7fb4c_isrForCrosstalkSources_946479_24.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/52b2f4f3-66cb-4317-9323-e15a1ce7fb4c_isrForCrosstalkSources_946479_24.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 3f9959331a0fed2b14a48f09595a9180 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e54686dc-1c6d-4d4a-aff0-8bc387f26195_isrForCrosstalkSources_946370_16.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e54686dc-1c6d-4d4a-aff0-8bc387f26195_isrForCrosstalkSources_946370_16.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 7c6ee0f3c0d5d2c580491d2eda7c2ead parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3498a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/943a0be2-701c-4eec-94b6-c62dce6a04a0_isrForCrosstalkSources_946369_55.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/943a0be2-701c-4eec-94b6-c62dce6a04a0_isrForCrosstalkSources_946369_55.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 7b449066ad04d4b1222967baceff94de parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/23874a99-7c0b-4579-aed8-5be01192485e_isrForCrosstalkSources_946371_54.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/23874a99-7c0b-4579-aed8-5be01192485e_isrForCrosstalkSources_946371_54.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 178d02f8c71133075de0cdcf7763b689 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3499e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/ad8de4ba-ee5a-4a25-8fc1-54cbf84802dd_isrForCrosstalkSources_946370_28.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/ad8de4ba-ee5a-4a25-8fc1-54cbf84802dd_isrForCrosstalkSources_946370_28.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash ed4dca0dae8a8c7fe79df23cc0a41020 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3496c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/16c360e6-285d-436f-ac30-a3f45abf88b8_isrForCrosstalkSources_946479_22.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/16c360e6-285d-436f-ac30-a3f45abf88b8_isrForCrosstalkSources_946479_22.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash d8035b7afd674f35d694a28d067287a0 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/cf895075-8ffd-4f0c-89b0-cb2f2ed863af_isrForCrosstalkSources_946369_5.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/cf895075-8ffd-4f0c-89b0-cb2f2ed863af_isrForCrosstalkSources_946369_5.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 5b21968f6d67f8b096d42cd56f37d955 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/89f02670-8602-4c78-90cd-ed7d67ec8862_isrForCrosstalkSources_946477_17.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/89f02670-8602-4c78-90cd-ed7d67ec8862_isrForCrosstalkSources_946477_17.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 8561022ae1b45414fbead8e86ec594f0 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/dc4af8c3-75bd-4798-81c6-f63800096751_isrForCrosstalkSources_946369_27.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/dc4af8c3-75bd-4798-81c6-f63800096751_isrForCrosstalkSources_946369_27.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash fd6cabf7a148fd0ed918e9402ba468b7 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 23:13:08 proc_lsst.multi:170 [INFO] [multi] job_id 17025208 proc_lsst.multi INFO: [multi] job_id 17025208 2024-03-13 23:13:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 23:13:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025208 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025208 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17025208 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-13 23:13:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-13 23:13:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 23:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/8075932a-9bf6-47b7-8e54-a25318225675_isrForCrosstalkSources_946478_22.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/8075932a-9bf6-47b7-8e54-a25318225675_isrForCrosstalkSources_946478_22.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 401e56550a5da6d755d67b40c9f0bc49 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/493150de-1be7-4067-b78d-4293badb890a_isrForCrosstalkSources_946369_49.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/493150de-1be7-4067-b78d-4293badb890a_isrForCrosstalkSources_946369_49.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 78e382fc9442c89d96a047e72aa230e7 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/f139f80b-4665-43ef-80f6-5a968cef2ca3_isrForCrosstalkSources_946479_35.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/f139f80b-4665-43ef-80f6-5a968cef2ca3_isrForCrosstalkSources_946479_35.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash a4442007222db8e4817dd399ae9440a3 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3491c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/c7128841-0228-44e4-a58b-78bceafb1d09_isrForCrosstalkSources_946371_49.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/c7128841-0228-44e4-a58b-78bceafb1d09_isrForCrosstalkSources_946371_49.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 56f3ad37d70f96652618e783bbe12c2e parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/40bf93f7-9323-45f6-a7ce-ed957f54849c_isrForCrosstalkSources_946369_20.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/40bf93f7-9323-45f6-a7ce-ed957f54849c_isrForCrosstalkSources_946369_20.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 71dca5654755b23f4e48ca336f99cd9b parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/757e5924-070a-47c6-aa93-a0a7ed170613_isrForCrosstalkSources_946371_18.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/757e5924-070a-47c6-aa93-a0a7ed170613_isrForCrosstalkSources_946371_18.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 8cf5e80f783dfd72c688843257771256 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/73326ac6-b5bd-46b7-8d55-96f23bcca694_isrForCrosstalkSources_946369_16.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/73326ac6-b5bd-46b7-8d55-96f23bcca694_isrForCrosstalkSources_946369_16.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 7a7fb49fefc55bb58d5ebc4172cacdff parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0d7eb296-0944-4f0f-b081-f87e42ccabf6_isrForCrosstalkSources_946371_61.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0d7eb296-0944-4f0f-b081-f87e42ccabf6_isrForCrosstalkSources_946371_61.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash a1682296de496dcad8d73b1c29323646 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/ef00a507-e04e-4df6-8280-963e5328cc88_isrForCrosstalkSources_946479_4.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/ef00a507-e04e-4df6-8280-963e5328cc88_isrForCrosstalkSources_946479_4.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 07f090ebba1dd68f5aabf19f130a6f77 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/150d6b5c-a5ef-409e-aa05-6f57728f3790_isrForCrosstalkSources_946478_61.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/150d6b5c-a5ef-409e-aa05-6f57728f3790_isrForCrosstalkSources_946478_61.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 8072820dc8c4f5807263072db9429019 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/b65a1809-d1ba-4607-8597-631b64315c22_isrForCrosstalkSources_946477_3.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/b65a1809-d1ba-4607-8597-631b64315c22_isrForCrosstalkSources_946477_3.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 1b14eae5ec4b62a45d1bfacf002895ba parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/19066285-7d7f-4ded-b9f6-76886416cd1b_isrForCrosstalkSources_946478_43.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/19066285-7d7f-4ded-b9f6-76886416cd1b_isrForCrosstalkSources_946478_43.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 3e7250b3e70074acb07575f7fcf6a921 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/ed0c0a4d-7e93-4671-80eb-1be7e54cb0cc_isrForCrosstalkSources_946477_29.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/ed0c0a4d-7e93-4671-80eb-1be7e54cb0cc_isrForCrosstalkSources_946477_29.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 07a54a18d14549daacc6ef03991efd53 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3ec51067-66d1-4a49-8cba-9dde6ae16238_isrForCrosstalkSources_946477_13.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3ec51067-66d1-4a49-8cba-9dde6ae16238_isrForCrosstalkSources_946477_13.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash a0a650af85e75f786af67d7586368963 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/46320173-9709-4f00-a0af-9699284a6c89_isrForCrosstalkSources_946477_49.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/46320173-9709-4f00-a0af-9699284a6c89_isrForCrosstalkSources_946477_49.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash a4ad39bec32722a3ff660f1f63041042 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/b9ff6af2-87da-45d5-88b0-c978977f05cb_isrForCrosstalkSources_946477_60.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/b9ff6af2-87da-45d5-88b0-c978977f05cb_isrForCrosstalkSources_946477_60.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 198423c6502331dcdef079be7246d888 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9d20ab43-126a-4582-96c3-c77a6fad106e_isrForCrosstalkSources_946477_26.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9d20ab43-126a-4582-96c3-c77a6fad106e_isrForCrosstalkSources_946477_26.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 966f1330d44fcdd50d5433ac7e0bbacb parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ec55e126-1ca9-4c86-9d19-b9b6bdc7e03b_isrForCrosstalkSources_946478_47.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ec55e126-1ca9-4c86-9d19-b9b6bdc7e03b_isrForCrosstalkSources_946478_47.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash a62363d4b06588cda926379bc2594eb3 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a721cfeb-78a0-480b-a044-1f2f9b467427_isrForCrosstalkSources_946371_2.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a721cfeb-78a0-480b-a044-1f2f9b467427_isrForCrosstalkSources_946371_2.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 18ff8746c3b2ba3d2afc65ec60975ccf parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2342a0e5-0f51-4dba-b889-685f38f41ed7_isrForCrosstalkSources_946369_23.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2342a0e5-0f51-4dba-b889-685f38f41ed7_isrForCrosstalkSources_946369_23.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 9d453f78eb29a2564417a18b62057b39 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1f729181-d75e-4aae-945f-c5e6820d6f40_isrForCrosstalkSources_946369_12.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1f729181-d75e-4aae-945f-c5e6820d6f40_isrForCrosstalkSources_946369_12.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 7596471f318f226ae3fa67d3d66348fc parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c16fbf04-c857-4b2b-9fa9-4c345813db30_isrForCrosstalkSources_946370_33.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c16fbf04-c857-4b2b-9fa9-4c345813db30_isrForCrosstalkSources_946370_33.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash fa88e4995a295c68332ffced2fc4e91d parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c55e9fe4-f220-4841-8c91-bab9c4131285_isrForCrosstalkSources_946370_49.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c55e9fe4-f220-4841-8c91-bab9c4131285_isrForCrosstalkSources_946370_49.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash c11e07b059461caa95763b14331fea67 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/d54c9870-7ad0-4173-ba6a-cd8e820728f1_isrForCrosstalkSources_946479_15.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/d54c9870-7ad0-4173-ba6a-cd8e820728f1_isrForCrosstalkSources_946479_15.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 3538d971fc8fa923eb84aa6e99298708 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/2fcf7b8d-9592-4023-b2ab-42e8c01ad3a1_isrForCrosstalkSources_946478_57.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/2fcf7b8d-9592-4023-b2ab-42e8c01ad3a1_isrForCrosstalkSources_946478_57.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 23b6eae0d64f43551132ae9b4b9572f3 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/a0f8da35-2411-4a6c-9b29-6319cc42c93d_isrForCrosstalkSources_946477_43.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/a0f8da35-2411-4a6c-9b29-6319cc42c93d_isrForCrosstalkSources_946477_43.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 16c453e73d9bc0d5468802bccde4b481 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/203c7f41-dfa6-4792-b97c-ddc47a626036_isrForCrosstalkSources_946371_39.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/203c7f41-dfa6-4792-b97c-ddc47a626036_isrForCrosstalkSources_946371_39.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 1e663b0813a6e40ecd5cc4dae1759e6f parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/46e53022-5a95-4b7b-a3e5-fd21e893c8e9_isrForCrosstalkSources_946371_9.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/46e53022-5a95-4b7b-a3e5-fd21e893c8e9_isrForCrosstalkSources_946371_9.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 9b3dc36149fb3dbff6f7d30f991b6298 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/107684cd-8865-4426-94f0-3a89f1e0b3a4_isrForCrosstalkSources_946371_1.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/107684cd-8865-4426-94f0-3a89f1e0b3a4_isrForCrosstalkSources_946371_1.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash a9d5074b138193580c415fa6cb750074 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/96fd7ecd-855c-4241-91a7-1a5e97cdcaab_isrForCrosstalkSources_946370_24.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/96fd7ecd-855c-4241-91a7-1a5e97cdcaab_isrForCrosstalkSources_946370_24.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash a8d143289f50f4c7472520b594e58346 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/6a8e2571-b8dc-4dff-a870-ed3dcc7cd4cb_isrForCrosstalkSources_946370_9.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/6a8e2571-b8dc-4dff-a870-ed3dcc7cd4cb_isrForCrosstalkSources_946370_9.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 2de7724d8d3840b9373372517b45a613 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/b1443ead-267b-44f4-9d24-53c685865d11_isrForCrosstalkSources_946371_58.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/b1443ead-267b-44f4-9d24-53c685865d11_isrForCrosstalkSources_946371_58.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 087743680d7cf9a8d092b05486389ffd parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/d8d082e3-a038-42c6-b2e0-9a7fc5f8a688_isrForCrosstalkSources_946370_41.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/d8d082e3-a038-42c6-b2e0-9a7fc5f8a688_isrForCrosstalkSources_946370_41.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 62b68de191b4b5c923e1dac6186bf681 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/9377d46a-47c4-41d4-95c1-b36039f0ae34_isrForCrosstalkSources_946371_30.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/9377d46a-47c4-41d4-95c1-b36039f0ae34_isrForCrosstalkSources_946371_30.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash c28b813a75b3e5a93eeb39a62ccc717b parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 23:13:08 proc_lsst.multi:170 [INFO] [multi] job_id 17025219 proc_lsst.multi INFO: [multi] job_id 17025219 2024-03-13 23:13:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 23:13:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025219 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025219 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17025219 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-13 23:13:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-13 23:13:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 23:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4411c94b-8a86-42aa-bce3-e65af3b65faa_isrForCrosstalkSources_946370_62.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4411c94b-8a86-42aa-bce3-e65af3b65faa_isrForCrosstalkSources_946370_62.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 6dbe514078adbb785d1a3a16e754020f parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ee96815f-ce6d-4d09-9732-2cabc7897c8c_isrForCrosstalkSources_946478_19.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ee96815f-ce6d-4d09-9732-2cabc7897c8c_isrForCrosstalkSources_946478_19.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 9350386febc29673f3c39599cca00b1a parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f66de3ff-c479-471d-ad26-f9eba3c363eb_isrForCrosstalkSources_946477_11.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f66de3ff-c479-471d-ad26-f9eba3c363eb_isrForCrosstalkSources_946477_11.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 86b022babf387003d829f9b5ab5ed2d8 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/02d5c662-0357-478c-b926-cc55bcae00b2_isrForCrosstalkSources_946370_59.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/02d5c662-0357-478c-b926-cc55bcae00b2_isrForCrosstalkSources_946370_59.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 5b0fd091aff81de2659570aff7c77818 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/943e975d-23df-4d78-b1e6-85d507f3b92f_isrForCrosstalkSources_946479_20.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/943e975d-23df-4d78-b1e6-85d507f3b92f_isrForCrosstalkSources_946479_20.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 12913f364f13ac7915287c6ae84eb45d parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0e0d504b-d8e0-4e08-91ad-5d0ea671fcd9_isrForCrosstalkSources_946369_26.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0e0d504b-d8e0-4e08-91ad-5d0ea671fcd9_isrForCrosstalkSources_946369_26.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 972e38af65fba38af28d3bfe42c6fb62 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e160c802-fc45-4402-aa27-647f311b8cd9_isrForCrosstalkSources_946479_32.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e160c802-fc45-4402-aa27-647f311b8cd9_isrForCrosstalkSources_946479_32.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash b956c537b5a8381efba8646715e48db9 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/adfe5258-e5fd-4941-bd66-a10af1ff72fe_isrForCrosstalkSources_946479_18.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/adfe5258-e5fd-4941-bd66-a10af1ff72fe_isrForCrosstalkSources_946479_18.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 381d33eb037882e4d655eca51f3f5849 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3eb306aa-7e6c-440f-b61f-c1c50b816cac_isrForCrosstalkSources_946479_13.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3eb306aa-7e6c-440f-b61f-c1c50b816cac_isrForCrosstalkSources_946479_13.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 956090925a7b290f53c614955fcf2d67 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5080a587-5cfd-481d-b69d-d937a879dc40_isrForCrosstalkSources_946369_15.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5080a587-5cfd-481d-b69d-d937a879dc40_isrForCrosstalkSources_946369_15.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 5f9128c90b094bbd5bb051d8323c60e8 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d439fdcc-7956-4b02-896f-46beb0224cfa_isrForCrosstalkSources_946478_52.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d439fdcc-7956-4b02-896f-46beb0224cfa_isrForCrosstalkSources_946478_52.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 9d48d978ebc2abe6cc49eed7a980267e parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/24901007-50be-49aa-85b7-77e80b5437e9_isrForCrosstalkSources_946478_38.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/24901007-50be-49aa-85b7-77e80b5437e9_isrForCrosstalkSources_946478_38.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 1e5fbcb9b33f13d025edb92812163946 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c4ea9b5e-08e2-4769-80b2-a90c3d70f9d6_isrForCrosstalkSources_946370_21.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c4ea9b5e-08e2-4769-80b2-a90c3d70f9d6_isrForCrosstalkSources_946370_21.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash cfd1baf1f84ccaa5c16f289059a057d6 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ff136d41-db11-4f25-95d6-c2269c7543ed_isrForCrosstalkSources_946478_53.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ff136d41-db11-4f25-95d6-c2269c7543ed_isrForCrosstalkSources_946478_53.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 78ea635b0b9a74bdff5d31d7546bec91 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/65045e18-5aeb-4398-bb5f-96375df9a87f_isrForCrosstalkSources_946477_31.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/65045e18-5aeb-4398-bb5f-96375df9a87f_isrForCrosstalkSources_946477_31.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash c884fb10fb9e1f9b05b3810be37c9e69 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c944ff9a-9be0-4a8f-aa66-567f7d8130b2_isrForCrosstalkSources_946370_12.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c944ff9a-9be0-4a8f-aa66-567f7d8130b2_isrForCrosstalkSources_946370_12.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 72fda2ad85984b019a29e23c71631570 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c7bfa312-37f0-4407-967c-cab8f4bc4cfe_isrForCrosstalkSources_946369_29.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c7bfa312-37f0-4407-967c-cab8f4bc4cfe_isrForCrosstalkSources_946369_29.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 8abe8e6f72521e7b965b9857ca579b32 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/bb8cb57a-900f-436c-bc4a-ebad93fbc1d9_isrForCrosstalkSources_946479_47.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/bb8cb57a-900f-436c-bc4a-ebad93fbc1d9_isrForCrosstalkSources_946479_47.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 1be254db775430ae397ad7e792066056 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a349d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f552c9ad-f064-43cc-8965-8868990caecd_isrForCrosstalkSources_946478_34.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f552c9ad-f064-43cc-8965-8868990caecd_isrForCrosstalkSources_946478_34.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 54599db3192529d3745c2923930f9e7d parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/5adc7c1e-1be2-491e-852a-7a11a7480c69_isrForCrosstalkSources_946370_18.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/5adc7c1e-1be2-491e-852a-7a11a7480c69_isrForCrosstalkSources_946370_18.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 58c55f383ca7087f69e01e204c545de5 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e3c58c92-78a5-4d76-90b3-ed512cd9199d_isrForCrosstalkSources_946370_11.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e3c58c92-78a5-4d76-90b3-ed512cd9199d_isrForCrosstalkSources_946370_11.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 61495bacf2a0878200e490bd57b5f4ba parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/73e50b04-da2e-48b4-b42e-1db50f41c7ce_isrForCrosstalkSources_946477_8.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/73e50b04-da2e-48b4-b42e-1db50f41c7ce_isrForCrosstalkSources_946477_8.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 7d9622b43925afbb24d364674eead87e parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/bfe18b99-ccf6-4952-8484-520478310adc_isrForCrosstalkSources_946371_21.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/bfe18b99-ccf6-4952-8484-520478310adc_isrForCrosstalkSources_946371_21.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash a68202499f200ad7ac53afa7364397d4 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/245f0652-0c6e-418c-be87-3c4664d9393c_isrForCrosstalkSources_946477_46.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/245f0652-0c6e-418c-be87-3c4664d9393c_isrForCrosstalkSources_946477_46.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash e0f203a8b1355c728e2f965643392e5a parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e7acee43-3b35-4a1d-a9f8-59f8fbf43c8f_isrForCrosstalkSources_946477_57.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e7acee43-3b35-4a1d-a9f8-59f8fbf43c8f_isrForCrosstalkSources_946477_57.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 24569e3dd3b483910a4d171d3a25992f parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/de4239e4-e95a-4945-9100-d1bed240cb57_isrForCrosstalkSources_946477_55.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/de4239e4-e95a-4945-9100-d1bed240cb57_isrForCrosstalkSources_946477_55.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 96781323d32c123f5f0d02228b528641 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3a7ba5dd-1f7d-46da-953a-abbb873646da_isrForCrosstalkSources_946477_53.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3a7ba5dd-1f7d-46da-953a-abbb873646da_isrForCrosstalkSources_946477_53.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 902692b391601fc0ba6312c7f2f56699 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/884d5dcb-d026-451c-a80d-36cbb2e9c4bc_isrForCrosstalkSources_946478_2.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/884d5dcb-d026-451c-a80d-36cbb2e9c4bc_isrForCrosstalkSources_946478_2.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 8023ed8bb0bfdfe5a3960f0d8d7b79e8 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/89850c1b-32b9-4b42-8edd-c667098c738e_isrForCrosstalkSources_946479_42.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/89850c1b-32b9-4b42-8edd-c667098c738e_isrForCrosstalkSources_946479_42.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash b8696d6ae1c7dad45da3dffb28af42d0 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a163ec42-2944-4154-ad66-f61a381d8af8_isrForCrosstalkSources_946478_11.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a163ec42-2944-4154-ad66-f61a381d8af8_isrForCrosstalkSources_946478_11.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 72ec0e4cb61b9538026b6e545e34c398 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/e4c6ba2d-2461-40f2-8835-8e01c5c2d5fe_isrForCrosstalkSources_946369_60.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/e4c6ba2d-2461-40f2-8835-8e01c5c2d5fe_isrForCrosstalkSources_946369_60.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash a5754f8f54bc6759f2a7041231150ed3 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/4941ba6d-554a-4321-a8b1-f3adcb4614bd_isrForCrosstalkSources_946479_50.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/4941ba6d-554a-4321-a8b1-f3adcb4614bd_isrForCrosstalkSources_946479_50.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 3bd4ffc22541ee4d7b52dac78158c976 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 23:13:08 proc_lsst.multi:170 [INFO] [multi] job_id 17025220 proc_lsst.multi INFO: [multi] job_id 17025220 2024-03-13 23:13:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 23:13:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025220 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025220 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17025220 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-13 23:13:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-13 23:13:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 23:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/3a1226b4-73ba-476c-8d6b-dc5316e6f26c_isrForCrosstalkSources_946369_59.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/3a1226b4-73ba-476c-8d6b-dc5316e6f26c_isrForCrosstalkSources_946369_59.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 57611ab4035bc9cb5fb000b53805d249 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/da4a0698-6e00-457a-ae1f-63be6ba1d1d4_isrForCrosstalkSources_946477_10.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/da4a0698-6e00-457a-ae1f-63be6ba1d1d4_isrForCrosstalkSources_946477_10.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash ed4c21ae0504d5e86f9816eff31f868e parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a348d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c2563919-8fb8-437d-91fa-88cc918c61c4_isrForCrosstalkSources_946369_43.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c2563919-8fb8-437d-91fa-88cc918c61c4_isrForCrosstalkSources_946369_43.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 46bdd7fff8e357b9728af7c0b474b30a parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a34ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0ce39571-32a2-45ae-9cdb-d72e42dd606a_isrForCrosstalkSources_946370_35.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0ce39571-32a2-45ae-9cdb-d72e42dd606a_isrForCrosstalkSources_946370_35.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash acfe884836470e3e93e9738a592277b6 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/191142ba-d40c-40b4-95b9-7349aed7383e_isrForCrosstalkSources_946369_14.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/191142ba-d40c-40b4-95b9-7349aed7383e_isrForCrosstalkSources_946369_14.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 8b93b891f4c2fa9d1e497b16655d86a9 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/cb93b289-70be-4aa6-950b-3a54e8146089_isrForCrosstalkSources_946371_45.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/cb93b289-70be-4aa6-950b-3a54e8146089_isrForCrosstalkSources_946371_45.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 9e29283e842352299d037de494ab186d parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/054623cd-b992-4e65-8810-06b8a0bfeac5_isrForCrosstalkSources_946477_22.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/054623cd-b992-4e65-8810-06b8a0bfeac5_isrForCrosstalkSources_946477_22.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 076700c5f16b4d1682b11ae7197359e7 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/a3ea80f3-a440-4b48-9f87-131b314acb8d_isrForCrosstalkSources_946370_15.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/a3ea80f3-a440-4b48-9f87-131b314acb8d_isrForCrosstalkSources_946370_15.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash c481241dace558e205477641ec2ccc67 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f6462c8e-8706-45c5-b8ab-26f0fa28ef92_isrForCrosstalkSources_946477_1.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f6462c8e-8706-45c5-b8ab-26f0fa28ef92_isrForCrosstalkSources_946477_1.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 2f5d650eb1f2ac1e03ae574a84810e22 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1916d08c-9de0-462b-8767-2903a8977cab_isrForCrosstalkSources_946370_30.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1916d08c-9de0-462b-8767-2903a8977cab_isrForCrosstalkSources_946370_30.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash f13d34764d080cedefab30183d24c7d9 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/06df898f-9c34-4ee2-9ae8-04998a22ef01_isrForCrosstalkSources_946369_9.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/06df898f-9c34-4ee2-9ae8-04998a22ef01_isrForCrosstalkSources_946369_9.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 3150e4eaa61c6340926f72daaf5a9d0f parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/bed16a6d-554a-440f-9f81-b22ad1654d75_isrForCrosstalkSources_946369_38.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/bed16a6d-554a-440f-9f81-b22ad1654d75_isrForCrosstalkSources_946369_38.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 40480f2f06b8187e24bd95038b0b0b2b parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/03903c5e-6d0c-424a-afe6-a921e44b93d0_isrForCrosstalkSources_946370_44.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/03903c5e-6d0c-424a-afe6-a921e44b93d0_isrForCrosstalkSources_946370_44.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 2b4ee988a11350d9973a7e84bfdd1dd8 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9390606e-ab8f-485e-b92b-8201aab65b0f_isrForCrosstalkSources_946479_38.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9390606e-ab8f-485e-b92b-8201aab65b0f_isrForCrosstalkSources_946479_38.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 640935de9c96c74dca4dcc9d5d9f90cc parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/82d10d84-e7b5-401b-b1b4-2517a6f53543_isrForCrosstalkSources_946477_54.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/82d10d84-e7b5-401b-b1b4-2517a6f53543_isrForCrosstalkSources_946477_54.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 6768d0ba3e97b3b2fef7bbd05e2588af parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/85998cdf-8d1f-4963-bfbb-a1ae1ef52b0a_isrForCrosstalkSources_946479_52.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/85998cdf-8d1f-4963-bfbb-a1ae1ef52b0a_isrForCrosstalkSources_946479_52.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash d734e3a0a7d96c3ebd7eed0edad01ce4 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/cedfc79a-6854-4ad6-a119-c23ef7e2a762_isrForCrosstalkSources_946477_50.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/cedfc79a-6854-4ad6-a119-c23ef7e2a762_isrForCrosstalkSources_946477_50.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash b3748b8acaab5c93ad23b77832291091 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/8ee8f177-9278-4a80-97ec-53ddbdf70229_isrForCrosstalkSources_946479_1.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/8ee8f177-9278-4a80-97ec-53ddbdf70229_isrForCrosstalkSources_946479_1.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 9b375a85c53c6a22d4ec120529f338e0 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/72e78781-a358-486f-a823-4e2694e94abc_isrForCrosstalkSources_946371_46.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/72e78781-a358-486f-a823-4e2694e94abc_isrForCrosstalkSources_946371_46.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash be7f603e1c5d44ccc6ca82bed4f42a9f parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/14e84241-1378-403c-82de-984954e19477_isrForCrosstalkSources_946479_17.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/14e84241-1378-403c-82de-984954e19477_isrForCrosstalkSources_946479_17.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 8e613732618c712ae1b96d2d3d042c00 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1973ce09-ba36-44d5-9ecc-3f76e64d6fb1_isrForCrosstalkSources_946369_54.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1973ce09-ba36-44d5-9ecc-3f76e64d6fb1_isrForCrosstalkSources_946369_54.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 87fd3b0b180d36232adff9acf9bf1b53 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/d23c061d-4f40-4268-895a-7b99f44cfaf9_isrForCrosstalkSources_946479_53.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/d23c061d-4f40-4268-895a-7b99f44cfaf9_isrForCrosstalkSources_946479_53.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 3a3b926b25fa396f25a466816f7cc4f5 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/070ffbe7-202a-43ff-beea-278992156675_isrForCrosstalkSources_946369_33.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/070ffbe7-202a-43ff-beea-278992156675_isrForCrosstalkSources_946369_33.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 653f6efead7e224dab59d4a12a6f9248 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c76e81aa-6eea-45e4-8925-0f1758202f20_isrForCrosstalkSources_946478_9.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c76e81aa-6eea-45e4-8925-0f1758202f20_isrForCrosstalkSources_946478_9.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash d800f9c6dfbe0b5b1850d4c5e3d6197b parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 23:13:09 proc_lsst.multi:170 [INFO] [multi] job_id 17025221 proc_lsst.multi INFO: [multi] job_id 17025221 2024-03-13 23:13:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025221 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025221 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17025221 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-13 23:13:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-13 23:13:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/b9ec6b07-6758-404d-919c-76ef92cbabea_isrForCrosstalkSources_946370_38.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/b9ec6b07-6758-404d-919c-76ef92cbabea_isrForCrosstalkSources_946370_38.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash ef64131f6ca7239f31c23b935aeeeabe parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0f848643-14a4-4c94-9141-ca99a5ac19df_isrForCrosstalkSources_946477_39.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0f848643-14a4-4c94-9141-ca99a5ac19df_isrForCrosstalkSources_946477_39.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 37710ae27f9a862409bc1a6fe68fb3a0 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e5f29498-a8b5-413e-b5b1-56a2540b5c59_isrForCrosstalkSources_946371_52.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e5f29498-a8b5-413e-b5b1-56a2540b5c59_isrForCrosstalkSources_946371_52.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash b39e2749265c59e70c0bd49aa4f2f7fa parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ed9cabcc-b410-4acc-b101-cccb358ce7ea_isrForCrosstalkSources_946478_32.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ed9cabcc-b410-4acc-b101-cccb358ce7ea_isrForCrosstalkSources_946478_32.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 6e7cb5e60eea27b566d42b0f60bae3a5 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/dd804b3f-4545-4a62-818f-800cee989b4b_isrForCrosstalkSources_946479_60.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/dd804b3f-4545-4a62-818f-800cee989b4b_isrForCrosstalkSources_946479_60.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 9730dadc13ad1fa87e2291ef1de27985 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/ce14f43b-e2cd-42c3-a23a-18177457304b_isrForCrosstalkSources_946477_34.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/ce14f43b-e2cd-42c3-a23a-18177457304b_isrForCrosstalkSources_946477_34.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 93aafdf3f3af1c135100b786e0b6a45e parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/318c7003-9a8d-4470-b4c6-eeb72aec031a_isrForCrosstalkSources_946370_1.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/318c7003-9a8d-4470-b4c6-eeb72aec031a_isrForCrosstalkSources_946370_1.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 8b46da9ba76bb20b12d813a057202847 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/b3d60411-bce8-4893-87cb-ce49c011f1e8_isrForCrosstalkSources_946479_12.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/b3d60411-bce8-4893-87cb-ce49c011f1e8_isrForCrosstalkSources_946479_12.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash b50265b860303b47407350f6b81961a3 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/48acea09-0831-4d98-853d-050a3d6989e5_isrForCrosstalkSources_946478_39.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/48acea09-0831-4d98-853d-050a3d6989e5_isrForCrosstalkSources_946478_39.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash e25af13471a7bab3c94d3d5c1e6e32d1 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6d0a3fa4-5c26-49a3-978f-d2ec17c15984_isrForCrosstalkSources_946477_7.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6d0a3fa4-5c26-49a3-978f-d2ec17c15984_isrForCrosstalkSources_946477_7.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 476a0ce445c765ca793e6279e822c651 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e33ed253-aa02-44fb-a12b-f67992d60eab_isrForCrosstalkSources_946479_46.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e33ed253-aa02-44fb-a12b-f67992d60eab_isrForCrosstalkSources_946479_46.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 2126518d67dcba773b9599e0ea45e641 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/581c508b-6bb2-4a2d-aa43-7d9963a31541_isrForCrosstalkSources_946370_43.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/581c508b-6bb2-4a2d-aa43-7d9963a31541_isrForCrosstalkSources_946370_43.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 2b0a8d1bcdc61ac3c69ffc87097716f1 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/1af49b2d-41c8-48c2-8e61-b7923a5d48ab_isrForCrosstalkSources_946478_14.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/1af49b2d-41c8-48c2-8e61-b7923a5d48ab_isrForCrosstalkSources_946478_14.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 19290ad96687c95873cd935f8814260a parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c9679584-3d91-48cf-bc11-0a9355b28024_isrForCrosstalkSources_946369_41.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c9679584-3d91-48cf-bc11-0a9355b28024_isrForCrosstalkSources_946369_41.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 2a0451d79540f897d02636937151b2d9 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e0075896-d184-477b-bcba-176d6edeeac8_isrForCrosstalkSources_946478_58.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e0075896-d184-477b-bcba-176d6edeeac8_isrForCrosstalkSources_946478_58.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 77e3d37a102d513df0140c6b95ee8d42 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/7fcb135a-af14-4962-99b9-0767e28403ac_isrForCrosstalkSources_946478_30.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/7fcb135a-af14-4962-99b9-0767e28403ac_isrForCrosstalkSources_946478_30.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 2166a85129269ceb1ad8da082b05c97d parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/69c8e6c5-aba9-41d6-91b4-6fcbfc16982e_isrForCrosstalkSources_946477_59.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/69c8e6c5-aba9-41d6-91b4-6fcbfc16982e_isrForCrosstalkSources_946477_59.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash c7957f64a9123ce4da2bf4da88cf89cb parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0f9d31dd-25df-4b11-885a-d1631751be6c_isrForCrosstalkSources_946371_5.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0f9d31dd-25df-4b11-885a-d1631751be6c_isrForCrosstalkSources_946371_5.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash c1467e3692fbe65ca296dee16626a5bf parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4fd9ae36-a266-4ee5-ab60-e87f23712fbf_isrForCrosstalkSources_946370_54.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4fd9ae36-a266-4ee5-ab60-e87f23712fbf_isrForCrosstalkSources_946370_54.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash f631dc93ae74fe5a3ce3145fefb29674 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/47877381-4542-4ff7-80a0-0989f9684f3b_isrForCrosstalkSources_946369_3.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/47877381-4542-4ff7-80a0-0989f9684f3b_isrForCrosstalkSources_946369_3.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 6f9209851cfba3def5093f66793c536c parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/ce98cb28-f91c-498f-9026-aa43885b0266_isrForCrosstalkSources_946369_8.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/ce98cb28-f91c-498f-9026-aa43885b0266_isrForCrosstalkSources_946369_8.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 55c28a05be9fc29cf14c4b2ae7bfb7f9 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4be83675-b42d-441e-829c-1c922d325e12_isrForCrosstalkSources_946371_26.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4be83675-b42d-441e-829c-1c922d325e12_isrForCrosstalkSources_946371_26.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 58dc8f771977a67d62ec47ce777c905e parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/72b41ea4-0f7f-4b17-9994-c60e3ac70e42_isrForCrosstalkSources_946479_9.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/72b41ea4-0f7f-4b17-9994-c60e3ac70e42_isrForCrosstalkSources_946479_9.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 532221735ec79e5439276b36d5b7e672 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e0d6ba7d-276d-48a7-bd8b-1465e039e9a5_isrForCrosstalkSources_946371_12.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e0d6ba7d-276d-48a7-bd8b-1465e039e9a5_isrForCrosstalkSources_946371_12.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 76e35a737decc5a9d7f7eb9e55d7da81 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/39686566-2492-44c0-9956-85faaf14190a_isrForCrosstalkSources_946478_48.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/39686566-2492-44c0-9956-85faaf14190a_isrForCrosstalkSources_946478_48.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash ea2eca1e4547ba0acd9075a10e4af5bb parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/b3cbb415-9c40-4825-9488-524bb6b3adc6_isrForCrosstalkSources_946479_41.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/b3cbb415-9c40-4825-9488-524bb6b3adc6_isrForCrosstalkSources_946479_41.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 50bb37740ec0cba810193e81c9448460 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/df1f2592-4f90-4e24-b86f-831a7d7155c6_isrForCrosstalkSources_946479_10.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/df1f2592-4f90-4e24-b86f-831a7d7155c6_isrForCrosstalkSources_946479_10.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 231d9744286e64cc55805bed825d3db5 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/92ac0362-bfc6-45da-b076-1e448dd8b02b_isrForCrosstalkSources_946478_29.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/92ac0362-bfc6-45da-b076-1e448dd8b02b_isrForCrosstalkSources_946478_29.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash afe3f3ca2c304f375f67d82fd75e3a41 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 23:13:09 proc_lsst.multi:170 [INFO] [multi] job_id 17025222 proc_lsst.multi INFO: [multi] job_id 17025222 2024-03-13 23:13:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 23:13:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025222 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025222 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17025222 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-13 23:13:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-13 23:13:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 23:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2eb50e37-d7c8-4e85-8439-9e8f590902f6_isrForCrosstalkSources_946477_56.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2eb50e37-d7c8-4e85-8439-9e8f590902f6_isrForCrosstalkSources_946477_56.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash ac06c02a791c96e5dc1d281ca5f0fc6c parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4b91b26b-0172-43d9-9dbe-b3de523d084f_isrForCrosstalkSources_946370_37.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4b91b26b-0172-43d9-9dbe-b3de523d084f_isrForCrosstalkSources_946370_37.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 7b04753fb6bb21aa277aa7fdce18d2ea parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/aec6474f-09e1-47a0-92cd-7d104b8d8200_isrForCrosstalkSources_946371_22.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/aec6474f-09e1-47a0-92cd-7d104b8d8200_isrForCrosstalkSources_946371_22.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 94f2c88b389f6e8ab4326b9c776cbae5 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c6ac568d-4119-4774-8701-d27fedc73b5c_isrForCrosstalkSources_946478_20.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c6ac568d-4119-4774-8701-d27fedc73b5c_isrForCrosstalkSources_946478_20.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash b69b3f6a6ab55deb0e428d91a4b7d959 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0aa0b2c3-3117-4e16-9dbd-ccdf8d62ce79_isrForCrosstalkSources_946369_18.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0aa0b2c3-3117-4e16-9dbd-ccdf8d62ce79_isrForCrosstalkSources_946369_18.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash cb5ddf7b21b650f241d0d8886c5553b7 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/de1c7f11-65e9-4722-aae1-4aec850c9df0_isrForCrosstalkSources_946370_13.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/de1c7f11-65e9-4722-aae1-4aec850c9df0_isrForCrosstalkSources_946370_13.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash c9e0b6d4359b8102e81f19a321298028 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a2620c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/572daf73-dd8d-47bd-88fc-db44375d113c_isrForCrosstalkSources_946369_47.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/572daf73-dd8d-47bd-88fc-db44375d113c_isrForCrosstalkSources_946369_47.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 0e24314ce4e4dcfa776fc77402f3bd6b parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a262200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6ba28200-cb11-488c-9292-2986a14c42c1_isrForCrosstalkSources_946371_42.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6ba28200-cb11-488c-9292-2986a14c42c1_isrForCrosstalkSources_946371_42.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash bc3fd0ff1caa5f019671869c8485ae58 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a262020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ab9bbd24-8d5c-4935-9127-8ec15b8ff56b_isrForCrosstalkSources_946371_37.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ab9bbd24-8d5c-4935-9127-8ec15b8ff56b_isrForCrosstalkSources_946371_37.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 03b09aac3fc1f878709ada863248d1c2 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a261f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0b8e9775-51f4-4d38-ab9d-671523f1457d_isrForCrosstalkSources_946369_51.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0b8e9775-51f4-4d38-ab9d-671523f1457d_isrForCrosstalkSources_946369_51.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 51483f16d9762dd1b9a53f5028fbc867 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a261ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/d5de5f7c-a337-47f2-ae9c-e673863caf82_isrForCrosstalkSources_946477_4.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/d5de5f7c-a337-47f2-ae9c-e673863caf82_isrForCrosstalkSources_946477_4.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 9ba6963265e007f036f1cd9c922a8bd3 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/de06b344-d3fc-4f87-895c-dae03ae7f2de_isrForCrosstalkSources_946371_17.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/de06b344-d3fc-4f87-895c-dae03ae7f2de_isrForCrosstalkSources_946371_17.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 1470c63c1a6a7c5e52150c836a8981cc parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/06238329-c8d4-41f4-8801-b41e3103d7ae_isrForCrosstalkSources_946477_51.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/06238329-c8d4-41f4-8801-b41e3103d7ae_isrForCrosstalkSources_946477_51.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 20c49c427971c2471ee0f496f89b926a parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5ce7c244-0ec1-4e2b-bcda-75a07013fde3_isrForCrosstalkSources_946369_17.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5ce7c244-0ec1-4e2b-bcda-75a07013fde3_isrForCrosstalkSources_946369_17.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 7c0cf0e295b0c45a513eed8e3139226e parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/78c8ef83-3960-4cec-9f3c-c3b534c4afd8_isrForCrosstalkSources_946477_36.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/78c8ef83-3960-4cec-9f3c-c3b534c4afd8_isrForCrosstalkSources_946477_36.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 0bd0c9ebb95a7cae265a153ae12594e7 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/dcfb5ad7-9d8a-457c-9b92-ffc652c63342_isrForCrosstalkSources_946371_34.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/dcfb5ad7-9d8a-457c-9b92-ffc652c63342_isrForCrosstalkSources_946371_34.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 231fd3afd090cd8948bb49982bcd5842 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/17c19e79-1a23-4b9f-95c4-c39ba0c75a5f_isrForCrosstalkSources_946477_45.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/17c19e79-1a23-4b9f-95c4-c39ba0c75a5f_isrForCrosstalkSources_946477_45.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash f4768023d39f696b929d234bda102459 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/33589ec4-7abd-4b30-841f-857ed4fb25d8_isrForCrosstalkSources_946370_10.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/33589ec4-7abd-4b30-841f-857ed4fb25d8_isrForCrosstalkSources_946370_10.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash c8382df44d02fd041648d54e2d064f8f parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/373279ca-4da7-4549-8c87-9b9c6cac8514_isrForCrosstalkSources_946370_58.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/373279ca-4da7-4549-8c87-9b9c6cac8514_isrForCrosstalkSources_946370_58.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash d084043d4e4093659a248ce7a8faf8ad parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/35f5059f-10e1-4cca-8652-8b3bbd68190e_isrForCrosstalkSources_946477_24.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/35f5059f-10e1-4cca-8652-8b3bbd68190e_isrForCrosstalkSources_946477_24.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash ae0408a48f6748a56aa110a18a7ce425 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9c0d45a0-83cd-4c1a-a57f-ae625a76267c_isrForCrosstalkSources_946479_28.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9c0d45a0-83cd-4c1a-a57f-ae625a76267c_isrForCrosstalkSources_946479_28.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash c01c66e7f24475efc5c2546c84b601ec parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/5c81372b-bcec-462e-8804-105d5a26c5ef_isrForCrosstalkSources_946477_37.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/5c81372b-bcec-462e-8804-105d5a26c5ef_isrForCrosstalkSources_946477_37.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash a405fbdee86f98223cbe933ded817ebd parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ed80366f-51aa-447e-9451-ee70b08cbb54_isrForCrosstalkSources_946371_35.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ed80366f-51aa-447e-9451-ee70b08cbb54_isrForCrosstalkSources_946371_35.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 5e04c5d658d7d187cc6ea4bdb31228aa parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/071180af-de8c-48fa-86dc-1cf06a4c87b1_isrForCrosstalkSources_946477_12.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/071180af-de8c-48fa-86dc-1cf06a4c87b1_isrForCrosstalkSources_946477_12.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 71d469a7390f6538fa779efa58e9b4b6 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/5ec308cb-2547-4858-af13-09282b021cab_isrForCrosstalkSources_946478_45.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/5ec308cb-2547-4858-af13-09282b021cab_isrForCrosstalkSources_946478_45.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash d1568864c0ee11d1d68617153fbebb77 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e420ae37-bc96-4521-8dd2-16cfaac8b648_isrForCrosstalkSources_946370_45.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e420ae37-bc96-4521-8dd2-16cfaac8b648_isrForCrosstalkSources_946370_45.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash a5473b02308c764a4498b9868903be4e parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/699773c6-7411-45db-8a10-cdb3b84b409a_isrForCrosstalkSources_946370_60.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/699773c6-7411-45db-8a10-cdb3b84b409a_isrForCrosstalkSources_946370_60.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash f93ff075237946383f87491a0dff878a parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/b43671ee-5f4d-4875-9728-b554d24653be_isrForCrosstalkSources_946371_47.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/b43671ee-5f4d-4875-9728-b554d24653be_isrForCrosstalkSources_946371_47.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 2cf4352d3b7afc1056164b5b9d78480c parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 23:13:09 proc_lsst.multi:170 [INFO] [multi] job_id 17025223 proc_lsst.multi INFO: [multi] job_id 17025223 2024-03-13 23:13:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 23:13:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025223 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025223 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17025223 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-13 23:13:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-13 23:13:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 23:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/ac76c08b-710a-41e3-8c45-72d02e4e3851_isrForCrosstalkSources_946370_32.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/ac76c08b-710a-41e3-8c45-72d02e4e3851_isrForCrosstalkSources_946370_32.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 8f0415515ccdb9d81fc568ab9d732950 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/f32d6e6e-cf11-412e-b706-8555cd17a5e9_isrForCrosstalkSources_946370_56.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/f32d6e6e-cf11-412e-b706-8555cd17a5e9_isrForCrosstalkSources_946370_56.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash e32e958896a4adf30a69598319288310 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/26ec77cd-4a58-4c4e-8553-4435fcf6fd1e_isrForCrosstalkSources_946479_33.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/26ec77cd-4a58-4c4e-8553-4435fcf6fd1e_isrForCrosstalkSources_946479_33.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash ffe5a8de99578d9c84dd0bf15b2d89bf parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a263100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/28ee0843-4b69-43d7-9ba6-418e01d410e4_isrForCrosstalkSources_946369_35.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/28ee0843-4b69-43d7-9ba6-418e01d410e4_isrForCrosstalkSources_946369_35.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 0f1ae523a27e838092ea5a204b2cab18 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a263060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/5a8feb52-3de2-4622-9556-4a0a4d1b9681_isrForCrosstalkSources_946371_24.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/5a8feb52-3de2-4622-9556-4a0a4d1b9681_isrForCrosstalkSources_946371_24.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 06f0dcad11da74b5a02068e39f910321 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a262fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ff7e6914-9461-4c95-ac79-ef59cf722abe_isrForCrosstalkSources_946371_19.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ff7e6914-9461-4c95-ac79-ef59cf722abe_isrForCrosstalkSources_946371_19.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 280823f817462e7c662cb4d8bd74b455 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c313b4ac-fe99-41d7-8483-ccaa368260c1_isrForCrosstalkSources_946479_8.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c313b4ac-fe99-41d7-8483-ccaa368260c1_isrForCrosstalkSources_946479_8.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash c8e5dfa97281487dc7a07ef2c8208771 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ece1ba95-2494-4fc3-946e-8db001af3163_isrForCrosstalkSources_946371_4.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ece1ba95-2494-4fc3-946e-8db001af3163_isrForCrosstalkSources_946371_4.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash d0ae244df63964e06b1630b1996c3782 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/620bb6be-b9f6-4925-b1f9-e964d8074e76_isrForCrosstalkSources_946371_57.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/620bb6be-b9f6-4925-b1f9-e964d8074e76_isrForCrosstalkSources_946371_57.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 3efc0ade6783d5291e9a566322bd83d7 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/fc31c5de-9e36-4071-91a7-f84fd169ebf2_isrForCrosstalkSources_946369_10.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/fc31c5de-9e36-4071-91a7-f84fd169ebf2_isrForCrosstalkSources_946369_10.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash bc061ea5dcd34e8fb8f68755d52e31c9 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e208afd9-fd45-42f2-b4e8-115d715ee801_isrForCrosstalkSources_946478_25.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e208afd9-fd45-42f2-b4e8-115d715ee801_isrForCrosstalkSources_946478_25.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash e390e7a9fd32b6640a51ec20a9983e62 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/6b41bb53-0658-4bff-8a9d-b9139fe7d7c1_isrForCrosstalkSources_946370_53.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/6b41bb53-0658-4bff-8a9d-b9139fe7d7c1_isrForCrosstalkSources_946370_53.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 5e4d6097aa3116ca6b4eec7fed623b55 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d09ab204-d420-430b-ae7b-e0d353281f5a_isrForCrosstalkSources_946478_17.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d09ab204-d420-430b-ae7b-e0d353281f5a_isrForCrosstalkSources_946478_17.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 96f168409d97afc6ee7f770caede7a97 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a3b6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e0a4edcc-dcba-4266-a788-f6a00251347f_isrForCrosstalkSources_946478_24.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e0a4edcc-dcba-4266-a788-f6a00251347f_isrForCrosstalkSources_946478_24.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 2a291c72a44c47b5e924a531749778d2 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a262de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a1f89a77-8564-4d95-bdce-f1ce9b0e4321_isrForCrosstalkSources_946371_13.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a1f89a77-8564-4d95-bdce-f1ce9b0e4321_isrForCrosstalkSources_946371_13.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 59fc8b3e49675beba8a852d7a416e255 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a261d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/bc48ae07-b84e-4227-82d7-c53927a8f68d_isrForCrosstalkSources_946479_29.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/bc48ae07-b84e-4227-82d7-c53927a8f68d_isrForCrosstalkSources_946479_29.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash c4fe3378e8bbb8b073cb548057e2e20d parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e42a260040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/2d503654-2dcc-4109-a697-1635321f17a4_isrForCrosstalkSources_946370_42.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/2d503654-2dcc-4109-a697-1635321f17a4_isrForCrosstalkSources_946370_42.stderr 2024-03-13 23:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17025224 proc_lsst.multi INFO: [multi] job_id 17025224 2024-03-13 23:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 23:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025224 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025224 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17025224 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-13 23:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-13 23:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 23:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17025225 proc_lsst.multi INFO: [multi] job_id 17025225 2024-03-13 23:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 23:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025225 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025225 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17025225 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-13 23:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-13 23:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 23:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17025226 proc_lsst.multi INFO: [multi] job_id 17025226 2024-03-13 23:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 23:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025226 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025226 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17025226 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-13 23:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-13 23:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 23:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17025227 proc_lsst.multi INFO: [multi] job_id 17025227 2024-03-13 23:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 23:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025227 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025227 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17025227 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-13 23:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-13 23:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 23:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:11 proc_lsst.multi:170 [INFO] [multi] job_id 17025228 proc_lsst.multi INFO: [multi] job_id 17025228 2024-03-13 23:13:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 23:13:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025228 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025228 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17025228 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-13 23:13:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-13 23:13:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 23:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:11 proc_lsst.multi:170 [INFO] [multi] job_id 17025238 proc_lsst.multi INFO: [multi] job_id 17025238 2024-03-13 23:13:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 23:13:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025238 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025238 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17025238 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-13 23:13:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-13 23:13:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 23:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:12 proc_lsst.multi:170 [INFO] [multi] job_id 17025239 proc_lsst.multi INFO: [multi] job_id 17025239 2024-03-13 23:13:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 23:13:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025239 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025239 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17025239 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-13 23:13:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-13 23:13:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 23:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:12 proc_lsst.multi:170 [INFO] [multi] job_id 17025240 proc_lsst.multi INFO: [multi] job_id 17025240 2024-03-13 23:13:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 23:13:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025240 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025240 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17025240 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-13 23:13:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-13 23:13:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 23:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:12 proc_lsst.multi:170 [INFO] [multi] job_id 17025241 proc_lsst.multi INFO: [multi] job_id 17025241 2024-03-13 23:13:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 23:13:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025241 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025241 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17025241 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-13 23:13:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-13 23:13:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 23:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:13 proc_lsst.multi:170 [INFO] [multi] job_id 17025242 proc_lsst.multi INFO: [multi] job_id 17025242 2024-03-13 23:13:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 23:13:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025242 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025242 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17025242 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-13 23:13:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:13 proc_lsst.multi:170 [INFO] [multi] job_id 17025243 proc_lsst.multi INFO: [multi] job_id 17025243 2024-03-13 23:13:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 23:13:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025243 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025243 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17025243 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-13 23:13:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:13 proc_lsst.multi:170 [INFO] [multi] job_id 17025244 proc_lsst.multi INFO: [multi] job_id 17025244 2024-03-13 23:13:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 23:13:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025244 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025244 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17025244 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-13 23:13:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:13 proc_lsst.multi:170 [INFO] [multi] job_id 17025245 proc_lsst.multi INFO: [multi] job_id 17025245 2024-03-13 23:13:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025245 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025245 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17025245 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-13 23:13:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:13 proc_lsst.multi:170 [INFO] [multi] job_id 17025246 proc_lsst.multi INFO: [multi] job_id 17025246 2024-03-13 23:13:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 23:13:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025246 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025246 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17025246 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-13 23:13:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:13 proc_lsst.multi:170 [INFO] [multi] job_id 17025247 proc_lsst.multi INFO: [multi] job_id 17025247 2024-03-13 23:13:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 23:13:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025247 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025247 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17025247 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-13 23:13:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 23:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:14 proc_lsst.multi:170 [INFO] [multi] job_id 17025248 proc_lsst.multi INFO: [multi] job_id 17025248 2024-03-13 23:13:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 23:13:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025248 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025248 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17025248 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-13 23:13:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-13 23:13:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 23:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:14 proc_lsst.multi:170 [INFO] [multi] job_id 17025249 proc_lsst.multi INFO: [multi] job_id 17025249 2024-03-13 23:13:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 23:13:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025249 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025249 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17025249 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-13 23:13:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-13 23:13:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 23:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:14 proc_lsst.multi:170 [INFO] [multi] job_id 17025250 proc_lsst.multi INFO: [multi] job_id 17025250 2024-03-13 23:13:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 23:13:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025250 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025250 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17025250 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-13 23:13:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-13 23:13:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 23:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:14 proc_lsst.multi:170 [INFO] [multi] job_id 17025251 proc_lsst.multi INFO: [multi] job_id 17025251 2024-03-13 23:13:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 23:13:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025251 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025251 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17025251 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-13 23:13:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-13 23:13:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 23:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:15 proc_lsst.multi:170 [INFO] [multi] job_id 17025252 proc_lsst.multi INFO: [multi] job_id 17025252 2024-03-13 23:13:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 23:13:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025252 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025252 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17025252 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-13 23:13:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-13 23:13:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 23:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:15 proc_lsst.multi:170 [INFO] [multi] job_id 17025253 proc_lsst.multi INFO: [multi] job_id 17025253 2024-03-13 23:13:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 23:13:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025253 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025253 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17025253 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-13 23:13:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-13 23:13:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 23:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:15 proc_lsst.multi:170 [INFO] [multi] job_id 17025254 proc_lsst.multi INFO: [multi] job_id 17025254 2024-03-13 23:13:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 23:13:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025254 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025254 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17025254 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-13 23:13:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-13 23:13:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 23:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:16 proc_lsst.multi:170 [INFO] [multi] job_id 17025255 proc_lsst.multi INFO: [multi] job_id 17025255 2024-03-13 23:13:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17025255 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17025255 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17025255 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-13 23:13:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-13 23:13:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 23:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:16 proc_lsst.multi:170 [INFO] [multi] job_id 17025256 proc_lsst.multi INFO: [multi] job_id 17025256 2024-03-13 23:13:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:16 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17025256 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17025256 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17025256 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-13 23:13:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-13 23:13:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:16 proc_lsst.multi:170 [INFO] [multi] job_id 17025257 proc_lsst.multi INFO: [multi] job_id 17025257 2024-03-13 23:13:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 23:13:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:16 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17025257 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17025257 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17025257 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-13 23:13:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-13 23:13:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 23:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:17 proc_lsst.multi:170 [INFO] [multi] job_id 17025258 proc_lsst.multi INFO: [multi] job_id 17025258 2024-03-13 23:13:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 23:13:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:17 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17025258 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17025258 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17025258 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-13 23:13:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:17 proc_lsst.multi:170 [INFO] [multi] job_id 17025259 proc_lsst.multi INFO: [multi] job_id 17025259 2024-03-13 23:13:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 23:13:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:17 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17025259 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17025259 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17025259 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-13 23:13:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:17 proc_lsst.multi:170 [INFO] [multi] job_id 17025260 proc_lsst.multi INFO: [multi] job_id 17025260 2024-03-13 23:13:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:17 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17025260 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17025260 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17025260 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-13 23:13:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 23:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:18 proc_lsst.multi:170 [INFO] [multi] job_id 17025261 proc_lsst.multi INFO: [multi] job_id 17025261 2024-03-13 23:13:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025261 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025261 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17025261 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-13 23:13:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:18 proc_lsst.multi:170 [INFO] [multi] job_id 17025262 proc_lsst.multi INFO: [multi] job_id 17025262 2024-03-13 23:13:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 23:13:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025262 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025262 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17025262 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 38, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/37 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 38 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 55 more blocks parsl.executors.status_handling INFO: Scaling out by 55 blocks parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-13 23:13:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 390c1aad28145d9300ed5462ca506472 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/18ffcc0a-2566-4666-80b7-624cadcdacce_isrForCrosstalkSources_946370_29.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/18ffcc0a-2566-4666-80b7-624cadcdacce_isrForCrosstalkSources_946370_29.stderr 2024-03-13 23:13:18 proc_lsst.multi:170 [INFO] [multi] job_id 17025263 proc_lsst.multi INFO: [multi] job_id 17025263 2024-03-13 23:13:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 23:13:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025263 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025263 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17025263 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-13 23:13:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:18 proc_lsst.multi:170 [INFO] [multi] job_id 17025264 proc_lsst.multi INFO: [multi] job_id 17025264 2024-03-13 23:13:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 23:13:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025264 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025264 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17025264 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-13 23:13:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 23:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61f310fe5d9797122ffb93e103ca7143 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/27683d21-d3fe-45af-8973-2803f86a828b_isrForCrosstalkSources_946479_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/27683d21-d3fe-45af-8973-2803f86a828b_isrForCrosstalkSources_946479_59.stderr 2024-03-13 23:13:19 proc_lsst.multi:170 [INFO] [multi] job_id 17025265 proc_lsst.multi INFO: [multi] job_id 17025265 2024-03-13 23:13:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025265 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025265 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17025265 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-13 23:13:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:19 proc_lsst.multi:170 [INFO] [multi] job_id 17025266 proc_lsst.multi INFO: [multi] job_id 17025266 2024-03-13 23:13:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 23:13:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025266 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025266 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17025266 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-13 23:13:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:19 proc_lsst.multi:170 [INFO] [multi] job_id 17025267 proc_lsst.multi INFO: [multi] job_id 17025267 2024-03-13 23:13:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 23:13:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025267 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025267 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17025267 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-13 23:13:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 23:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:20 proc_lsst.multi:170 [INFO] [multi] job_id 17025268 proc_lsst.multi INFO: [multi] job_id 17025268 2024-03-13 23:13:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 23:13:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025268 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025268 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17025268 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-13 23:13:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:20 proc_lsst.multi:170 [INFO] [multi] job_id 17025269 proc_lsst.multi INFO: [multi] job_id 17025269 2024-03-13 23:13:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 23:13:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025269 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025269 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17025269 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-13 23:13:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:20 proc_lsst.multi:170 [INFO] [multi] job_id 17025270 proc_lsst.multi INFO: [multi] job_id 17025270 2024-03-13 23:13:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 23:13:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025270 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025270 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17025270 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-13 23:13:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:20 proc_lsst.multi:170 [INFO] [multi] job_id 17025271 proc_lsst.multi INFO: [multi] job_id 17025271 2024-03-13 23:13:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 23:13:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025271 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025271 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17025271 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-13 23:13:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 23:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:21 proc_lsst.multi:170 [INFO] [multi] job_id 17025272 proc_lsst.multi INFO: [multi] job_id 17025272 2024-03-13 23:13:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 23:13:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025272 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025272 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17025272 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-13 23:13:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:21 proc_lsst.multi:170 [INFO] [multi] job_id 17025273 proc_lsst.multi INFO: [multi] job_id 17025273 2024-03-13 23:13:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 23:13:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025273 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025273 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17025273 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-13 23:13:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:21 proc_lsst.multi:170 [INFO] [multi] job_id 17025274 proc_lsst.multi INFO: [multi] job_id 17025274 2024-03-13 23:13:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 23:13:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025274 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025274 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17025274 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-13 23:13:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 23:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:22 proc_lsst.multi:170 [INFO] [multi] job_id 17025275 proc_lsst.multi INFO: [multi] job_id 17025275 2024-03-13 23:13:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 23:13:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025275 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025275 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17025275 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-13 23:13:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-13 23:13:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 23:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:22 proc_lsst.multi:170 [INFO] [multi] job_id 17025276 proc_lsst.multi INFO: [multi] job_id 17025276 2024-03-13 23:13:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 23:13:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025276 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025276 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17025276 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-13 23:13:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-13 23:13:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 23:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17025277 proc_lsst.multi INFO: [multi] job_id 17025277 2024-03-13 23:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 23:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025277 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025277 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17025277 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-13 23:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17025278 proc_lsst.multi INFO: [multi] job_id 17025278 2024-03-13 23:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 23:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025278 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025278 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17025278 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-13 23:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17025279 proc_lsst.multi INFO: [multi] job_id 17025279 2024-03-13 23:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 23:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025279 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025279 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17025279 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-13 23:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17025280 proc_lsst.multi INFO: [multi] job_id 17025280 2024-03-13 23:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17025280 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17025280 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17025280 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-13 23:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17025281 proc_lsst.multi INFO: [multi] job_id 17025281 2024-03-13 23:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025281 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025281 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17025281 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-13 23:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:24 proc_lsst.multi:170 [INFO] [multi] job_id 17025282 proc_lsst.multi INFO: [multi] job_id 17025282 2024-03-13 23:13:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 23:13:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025282 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025282 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17025282 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-13 23:13:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:24 proc_lsst.multi:170 [INFO] [multi] job_id 17025283 proc_lsst.multi INFO: [multi] job_id 17025283 2024-03-13 23:13:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 23:13:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025283 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025283 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17025283 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-13 23:13:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:24 proc_lsst.multi:170 [INFO] [multi] job_id 17025284 proc_lsst.multi INFO: [multi] job_id 17025284 2024-03-13 23:13:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 23:13:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025284 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025284 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17025284 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-13 23:13:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:24 proc_lsst.multi:170 [INFO] [multi] job_id 17025285 proc_lsst.multi INFO: [multi] job_id 17025285 2024-03-13 23:13:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025285 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025285 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17025285 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-13 23:13:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:24 proc_lsst.multi:170 [INFO] [multi] job_id 17025286 proc_lsst.multi INFO: [multi] job_id 17025286 2024-03-13 23:13:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 23:13:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025286 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025286 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17025286 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-13 23:13:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 23:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:25 proc_lsst.multi:170 [INFO] [multi] job_id 17025287 proc_lsst.multi INFO: [multi] job_id 17025287 2024-03-13 23:13:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 23:13:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025287 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025287 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17025287 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-13 23:13:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:25 proc_lsst.multi:170 [INFO] [multi] job_id 17025288 proc_lsst.multi INFO: [multi] job_id 17025288 2024-03-13 23:13:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 23:13:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025288 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025288 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17025288 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-13 23:13:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:25 proc_lsst.multi:170 [INFO] [multi] job_id 17025289 proc_lsst.multi INFO: [multi] job_id 17025289 2024-03-13 23:13:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 23:13:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025289 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025289 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17025289 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-13 23:13:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 23:13:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:26 proc_lsst.multi:170 [INFO] [multi] job_id 17025290 proc_lsst.multi INFO: [multi] job_id 17025290 2024-03-13 23:13:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 23:13:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025290 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025290 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17025290 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-13 23:13:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:26 proc_lsst.multi:170 [INFO] [multi] job_id 17025291 proc_lsst.multi INFO: [multi] job_id 17025291 2024-03-13 23:13:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 23:13:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025291 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025291 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17025291 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-13 23:13:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:26 proc_lsst.multi:170 [INFO] [multi] job_id 17025292 proc_lsst.multi INFO: [multi] job_id 17025292 2024-03-13 23:13:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 23:13:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025292 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025292 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17025292 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-13 23:13:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 23:13:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:27 proc_lsst.multi:170 [INFO] [multi] job_id 17025293 proc_lsst.multi INFO: [multi] job_id 17025293 2024-03-13 23:13:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 23:13:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025293 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025293 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17025293 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-13 23:13:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cc0c6e47fab68cb384ea55b506438b0 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2d1af336-7710-4c5b-bd0f-e35a21523da3_isrForCrosstalkSources_946369_25.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2d1af336-7710-4c5b-bd0f-e35a21523da3_isrForCrosstalkSources_946369_25.stderr 2024-03-13 23:13:27 proc_lsst.multi:170 [INFO] [multi] job_id 17025294 proc_lsst.multi INFO: [multi] job_id 17025294 2024-03-13 23:13:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 23:13:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025294 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025294 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17025294 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-13 23:13:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:27 proc_lsst.multi:170 [INFO] [multi] job_id 17025295 proc_lsst.multi INFO: [multi] job_id 17025295 2024-03-13 23:13:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 23:13:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025295 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025295 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17025295 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-13 23:13:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 23:13:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:28 proc_lsst.multi:170 [INFO] [multi] job_id 17025296 proc_lsst.multi INFO: [multi] job_id 17025296 2024-03-13 23:13:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 23:13:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025296 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025296 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17025296 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-13 23:13:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:28 proc_lsst.multi:170 [INFO] [multi] job_id 17025297 proc_lsst.multi INFO: [multi] job_id 17025297 2024-03-13 23:13:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 23:13:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025297 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025297 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17025297 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-13 23:13:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:28 proc_lsst.multi:170 [INFO] [multi] job_id 17025298 proc_lsst.multi INFO: [multi] job_id 17025298 2024-03-13 23:13:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 23:13:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025298 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025298 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17025298 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-13 23:13:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f21df2237811b99a6561e3e05efb975d with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/9293570c-aeb6-477d-aa7e-69837bea5c6d_isrForCrosstalkSources_946478_31.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/9293570c-aeb6-477d-aa7e-69837bea5c6d_isrForCrosstalkSources_946478_31.stderr 2024-03-13 23:13:28 proc_lsst.multi:170 [INFO] [multi] job_id 17025299 proc_lsst.multi INFO: [multi] job_id 17025299 2024-03-13 23:13:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 23:13:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025299 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025299 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17025299 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-13 23:13:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 23:13:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:29 proc_lsst.multi:170 [INFO] [multi] job_id 17025300 proc_lsst.multi INFO: [multi] job_id 17025300 2024-03-13 23:13:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025300 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025300 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17025300 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-13 23:13:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:29 proc_lsst.multi:170 [INFO] [multi] job_id 17025301 proc_lsst.multi INFO: [multi] job_id 17025301 2024-03-13 23:13:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 23:13:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025301 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025301 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17025301 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-13 23:13:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:29 proc_lsst.multi:170 [INFO] [multi] job_id 17025302 proc_lsst.multi INFO: [multi] job_id 17025302 2024-03-13 23:13:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 23:13:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025302 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025302 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17025302 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-13 23:13:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 23:13:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:30 proc_lsst.multi:170 [INFO] [multi] job_id 17025303 proc_lsst.multi INFO: [multi] job_id 17025303 2024-03-13 23:13:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 23:13:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025303 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025303 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17025303 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-13 23:13:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:30 proc_lsst.multi:170 [INFO] [multi] job_id 17025304 proc_lsst.multi INFO: [multi] job_id 17025304 2024-03-13 23:13:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 23:13:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025304 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025304 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17025304 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-13 23:13:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:30 proc_lsst.multi:170 [INFO] [multi] job_id 17025305 proc_lsst.multi INFO: [multi] job_id 17025305 2024-03-13 23:13:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 23:13:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025305 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025305 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17025305 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-13 23:13:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 23:13:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:30 proc_lsst.multi:170 [INFO] [multi] job_id 17025306 proc_lsst.multi INFO: [multi] job_id 17025306 2024-03-13 23:13:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 23:13:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025306 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025306 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17025306 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-13 23:13:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:31 proc_lsst.multi:170 [INFO] [multi] job_id 17025307 proc_lsst.multi INFO: [multi] job_id 17025307 2024-03-13 23:13:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 23:13:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025307 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025307 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17025307 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-13 23:13:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:31 proc_lsst.multi:170 [INFO] [multi] job_id 17025308 proc_lsst.multi INFO: [multi] job_id 17025308 2024-03-13 23:13:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 23:13:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025308 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025308 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17025308 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-13 23:13:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:31 proc_lsst.multi:170 [INFO] [multi] job_id 17025309 proc_lsst.multi INFO: [multi] job_id 17025309 2024-03-13 23:13:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 23:13:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025309 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025309 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17025309 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-13 23:13:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:31 proc_lsst.multi:170 [INFO] [multi] job_id 17025310 proc_lsst.multi INFO: [multi] job_id 17025310 2024-03-13 23:13:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025310 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025310 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17025310 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-13 23:13:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:32 proc_lsst.multi:170 [INFO] [multi] job_id 17025311 proc_lsst.multi INFO: [multi] job_id 17025311 2024-03-13 23:13:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-13 23:13:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025311 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025311 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17025311 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-13 23:13:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:32 proc_lsst.multi:170 [INFO] [multi] job_id 17025312 proc_lsst.multi INFO: [multi] job_id 17025312 2024-03-13 23:13:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-13 23:13:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025312 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025312 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17025312 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-13 23:13:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:32 proc_lsst.multi:170 [INFO] [multi] job_id 17025313 proc_lsst.multi INFO: [multi] job_id 17025313 2024-03-13 23:13:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-13 23:13:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025313 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025313 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17025313 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-13 23:13:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-13 23:13:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:33 proc_lsst.multi:170 [INFO] [multi] job_id 17025314 proc_lsst.multi INFO: [multi] job_id 17025314 2024-03-13 23:13:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-13 23:13:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025314 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025314 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17025314 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-13 23:13:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:33 proc_lsst.multi:170 [INFO] [multi] job_id 17025315 proc_lsst.multi INFO: [multi] job_id 17025315 2024-03-13 23:13:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-13 23:13:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025315 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025315 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17025315 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-13 23:13:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:33 proc_lsst.multi:170 [INFO] [multi] job_id 17025316 proc_lsst.multi INFO: [multi] job_id 17025316 2024-03-13 23:13:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-13 23:13:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025316 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025316 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17025316 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-13 23:13:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 23:13:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 23:13:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-13 23:13:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.129.9,127.0.0.1,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54532 --result_port=54842 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 23:13:34 proc_lsst.multi:170 [INFO] [multi] job_id 17025317 proc_lsst.multi INFO: [multi] job_id 17025317 2024-03-13 23:13:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-13 23:13:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 23:13:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17025317 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17025317 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17025317 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 368 parsl.dataflow.strategy DEBUG: Executor multi has 368 active tasks, 1/92 running/pending blocks, and 65 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd35a0a8741b82b4fc97fa0246770f36 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/dc3f0df5-edfc-4601-be7c-0d618c9670a4_isrForCrosstalkSources_946479_31.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/dc3f0df5-edfc-4601-be7c-0d618c9670a4_isrForCrosstalkSources_946479_31.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3be5390680ab975e7f2a1238dceafdb with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/15e59511-2ec8-47fd-a48d-52c3371f1138_isrForCrosstalkSources_946370_26.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/15e59511-2ec8-47fd-a48d-52c3371f1138_isrForCrosstalkSources_946370_26.stderr 2024-03-13 23:13:39 proc_lsst.multi:146 [INFO] found job 20175 in provider local proc_lsst.multi INFO: found job 20175 in provider local 2024-03-13 23:13:39 proc_lsst.multi:146 [INFO] found job 17025207 in provider astro proc_lsst.multi INFO: found job 17025207 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:39 proc_lsst.multi:146 [INFO] found job 17025208 in provider astro proc_lsst.multi INFO: found job 17025208 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:39 proc_lsst.multi:146 [INFO] found job 17025219 in provider astro proc_lsst.multi INFO: found job 17025219 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:40 proc_lsst.multi:146 [INFO] found job 17025220 in provider astro proc_lsst.multi INFO: found job 17025220 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:40 proc_lsst.multi:146 [INFO] found job 17025221 in provider astro proc_lsst.multi INFO: found job 17025221 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:41 proc_lsst.multi:146 [INFO] found job 17025222 in provider astro proc_lsst.multi INFO: found job 17025222 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:41 proc_lsst.multi:146 [INFO] found job 17025223 in provider astro proc_lsst.multi INFO: found job 17025223 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:41 proc_lsst.multi:146 [INFO] found job 17025224 in provider astro proc_lsst.multi INFO: found job 17025224 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:42 proc_lsst.multi:146 [INFO] found job 17025225 in provider astro proc_lsst.multi INFO: found job 17025225 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:42 proc_lsst.multi:146 [INFO] found job 17025226 in provider astro proc_lsst.multi INFO: found job 17025226 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:42 proc_lsst.multi:146 [INFO] found job 17025227 in provider astro proc_lsst.multi INFO: found job 17025227 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:43 proc_lsst.multi:146 [INFO] found job 17025228 in provider astro proc_lsst.multi INFO: found job 17025228 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:43 proc_lsst.multi:146 [INFO] found job 17025238 in provider astro proc_lsst.multi INFO: found job 17025238 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:43 proc_lsst.multi:146 [INFO] found job 17025239 in provider astro proc_lsst.multi INFO: found job 17025239 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:44 proc_lsst.multi:146 [INFO] found job 17025240 in provider astro proc_lsst.multi INFO: found job 17025240 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e3acfab16766d0458003eed92d3e943 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/500c8c3a-215e-41cb-a5b1-af6134d3c393_isrForCrosstalkSources_946479_23.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/500c8c3a-215e-41cb-a5b1-af6134d3c393_isrForCrosstalkSources_946479_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:44 proc_lsst.multi:146 [INFO] found job 17025241 in provider astro proc_lsst.multi INFO: found job 17025241 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:44 proc_lsst.multi:146 [INFO] found job 17025242 in provider astro proc_lsst.multi INFO: found job 17025242 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:45 proc_lsst.multi:146 [INFO] found job 17025243 in provider astro proc_lsst.multi INFO: found job 17025243 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a1649b54f6fe0ff6281e6a9171fe658 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4702096b-ea52-45de-8cf3-006ffb04096e_isrForCrosstalkSources_946369_6.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4702096b-ea52-45de-8cf3-006ffb04096e_isrForCrosstalkSources_946369_6.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 158fd010a695770ad0c30cb2affce91f with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/f1e4df14-855f-4133-9baa-41d3a14ab688_isrForCrosstalkSources_946371_6.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/f1e4df14-855f-4133-9baa-41d3a14ab688_isrForCrosstalkSources_946371_6.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ff96c6b5af3359ea50f690b3086b7e0 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4faee463-19ef-4557-8797-e325a65b6567_isrForCrosstalkSources_946370_40.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4faee463-19ef-4557-8797-e325a65b6567_isrForCrosstalkSources_946370_40.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b0784bdab057a7fd10da0d5de5043a2 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/2b19b237-a3a9-47ac-97c0-e4262432aab7_isrForCrosstalkSources_946479_51.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/2b19b237-a3a9-47ac-97c0-e4262432aab7_isrForCrosstalkSources_946479_51.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8138950c5e77bd0c765e9224bc284ae5 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/45b2b833-4f0f-4c26-8b78-4b47082f9efb_isrForCrosstalkSources_946371_27.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/45b2b833-4f0f-4c26-8b78-4b47082f9efb_isrForCrosstalkSources_946371_27.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 798fbe063141c4c6cd1cc6fddcdf403f with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/02a74268-0989-4c9f-994e-8754a216c399_isrForCrosstalkSources_946370_19.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/02a74268-0989-4c9f-994e-8754a216c399_isrForCrosstalkSources_946370_19.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32ac3b10411162bf222fadf1e9090b39 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c6786af6-cd2d-417c-94e1-babb04db840a_isrForCrosstalkSources_946369_36.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c6786af6-cd2d-417c-94e1-babb04db840a_isrForCrosstalkSources_946369_36.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e353f533e56e2e723cce370cdcecf6c2 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e4b4c35e-b586-4a9f-a54a-8006538f9ae1_isrForCrosstalkSources_946477_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e4b4c35e-b586-4a9f-a54a-8006538f9ae1_isrForCrosstalkSources_946477_61.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff4e2affe66e306955f543b5a5d3a044 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/5401da2d-9f29-4a8a-88d5-28104bb18be1_isrForCrosstalkSources_946479_58.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/5401da2d-9f29-4a8a-88d5-28104bb18be1_isrForCrosstalkSources_946479_58.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 581df1859afc9ae64237f09456f5b4a1 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/83e612a6-0868-436e-8cc9-fee9a3d89ca7_isrForCrosstalkSources_946479_55.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/83e612a6-0868-436e-8cc9-fee9a3d89ca7_isrForCrosstalkSources_946479_55.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 053f8a24ef73539fc8d3b9f0b2e734e2 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/cf5eb18e-8bf4-47c7-b862-683a58c1b9a7_isrForCrosstalkSources_946478_4.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/cf5eb18e-8bf4-47c7-b862-683a58c1b9a7_isrForCrosstalkSources_946478_4.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5164eab51d954dd1d99fc4c0506d942 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/17151c56-a534-48c0-a781-13091ed19211_isrForCrosstalkSources_946369_48.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/17151c56-a534-48c0-a781-13091ed19211_isrForCrosstalkSources_946369_48.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abded8e68975037d9f52517f564585cf with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/066db2ee-4e0b-493b-a6f3-8a3f943a5e38_isrForCrosstalkSources_946369_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/066db2ee-4e0b-493b-a6f3-8a3f943a5e38_isrForCrosstalkSources_946369_42.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a2836b3c46cea33fdf74d3fe1d014f4 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/1f6c0c67-81f6-48cc-b9f4-86ea56a4c1e5_isrForCrosstalkSources_946478_50.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/1f6c0c67-81f6-48cc-b9f4-86ea56a4c1e5_isrForCrosstalkSources_946478_50.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5c59bced54b83c1fc348aa47b54a15f with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e3885461-e6b3-444e-86bd-a1444bbce5a7_isrForCrosstalkSources_946371_28.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e3885461-e6b3-444e-86bd-a1444bbce5a7_isrForCrosstalkSources_946371_28.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51e74ec8ba6b5cbe47e6d28af0c93cec with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/30cdf018-0e5a-467c-b886-a9d40f121c7d_isrForCrosstalkSources_946369_52.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/30cdf018-0e5a-467c-b886-a9d40f121c7d_isrForCrosstalkSources_946369_52.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35bf92b26982bae6baa0e32f59236cef with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6cbd0cda-6fb5-4476-a576-b4860b0ecc80_isrForCrosstalkSources_946477_27.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6cbd0cda-6fb5-4476-a576-b4860b0ecc80_isrForCrosstalkSources_946477_27.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 659f42c433c4a1f3d42ff8331fff2d77 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/a6e19a6e-c87b-4a08-9c71-3257b6f83fa8_isrForCrosstalkSources_946477_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/a6e19a6e-c87b-4a08-9c71-3257b6f83fa8_isrForCrosstalkSources_946477_44.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa4775ec808116dbdcb777482b2e91e0 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/2ef60bc2-5357-410c-993d-c42a178cf7b0_isrForCrosstalkSources_946371_48.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/2ef60bc2-5357-410c-993d-c42a178cf7b0_isrForCrosstalkSources_946371_48.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea8a728860f2682a806303e3d0ccc7d with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9d871c65-78fe-40a2-9cb5-229d53daab40_isrForCrosstalkSources_946477_5.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9d871c65-78fe-40a2-9cb5-229d53daab40_isrForCrosstalkSources_946477_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:45 proc_lsst.multi:146 [INFO] found job 17025244 in provider astro proc_lsst.multi INFO: found job 17025244 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dee3371fdfd0e37e225700b632fe9ae3 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/0e1a4e91-b3c0-4e2b-b892-d6ae189ca0c8_isrForCrosstalkSources_946479_57.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/0e1a4e91-b3c0-4e2b-b892-d6ae189ca0c8_isrForCrosstalkSources_946479_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:45 proc_lsst.multi:146 [INFO] found job 17025245 in provider astro proc_lsst.multi INFO: found job 17025245 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:45 proc_lsst.multi:146 [INFO] found job 17025246 in provider astro proc_lsst.multi INFO: found job 17025246 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:46 proc_lsst.multi:146 [INFO] found job 17025247 in provider astro proc_lsst.multi INFO: found job 17025247 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:46 proc_lsst.multi:146 [INFO] found job 17025248 in provider astro proc_lsst.multi INFO: found job 17025248 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:47 proc_lsst.multi:146 [INFO] found job 17025249 in provider astro proc_lsst.multi INFO: found job 17025249 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:47 proc_lsst.multi:146 [INFO] found job 17025250 in provider astro proc_lsst.multi INFO: found job 17025250 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:47 proc_lsst.multi:146 [INFO] found job 17025251 in provider astro proc_lsst.multi INFO: found job 17025251 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:48 proc_lsst.multi:146 [INFO] found job 17025252 in provider astro proc_lsst.multi INFO: found job 17025252 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:48 proc_lsst.multi:146 [INFO] found job 17025253 in provider astro proc_lsst.multi INFO: found job 17025253 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:48 proc_lsst.multi:146 [INFO] found job 17025254 in provider astro proc_lsst.multi INFO: found job 17025254 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:49 proc_lsst.multi:146 [INFO] found job 17025255 in provider astro proc_lsst.multi INFO: found job 17025255 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:49 proc_lsst.multi:146 [INFO] found job 17025256 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025256 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:49 proc_lsst.multi:146 [INFO] found job 17025257 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025257 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:50 proc_lsst.multi:146 [INFO] found job 17025258 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025258 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:50 proc_lsst.multi:146 [INFO] found job 17025259 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025259 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:51 proc_lsst.multi:146 [INFO] found job 17025260 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025260 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb6f239e0902c388cf1f2a8ce7c5a677 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f98b2c2b-a12f-43aa-a652-7d62f27bab3e_isrForCrosstalkSources_946477_18.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f98b2c2b-a12f-43aa-a652-7d62f27bab3e_isrForCrosstalkSources_946477_18.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b3f6df0a62d02f4116ee1471697644f with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/dd0cab6d-8b77-4825-b363-27f4cec77740_isrForCrosstalkSources_946371_3.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/dd0cab6d-8b77-4825-b363-27f4cec77740_isrForCrosstalkSources_946371_3.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f975bdac4cf697b15dcfd6cdaf5b2496 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d7cfeac9-4fc5-4a82-b915-22a96f5b348a_isrForCrosstalkSources_946371_44.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d7cfeac9-4fc5-4a82-b915-22a96f5b348a_isrForCrosstalkSources_946371_44.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0671872d3e1341d56df16118098fd0cf with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6e2d3f32-f7cc-4192-897b-69ee0f272c13_isrForCrosstalkSources_946371_31.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6e2d3f32-f7cc-4192-897b-69ee0f272c13_isrForCrosstalkSources_946371_31.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdb53b1499ba11e4845c1b736d87a26b with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/44bdc3a2-63dd-4355-9dd2-7dbc64cf95ff_isrForCrosstalkSources_946479_21.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/44bdc3a2-63dd-4355-9dd2-7dbc64cf95ff_isrForCrosstalkSources_946479_21.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e42330b5fdeb8591392a84a27edf87c9 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6921b943-6ce9-43e9-ac9b-caf10068096a_isrForCrosstalkSources_946371_55.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6921b943-6ce9-43e9-ac9b-caf10068096a_isrForCrosstalkSources_946371_55.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a0056f14c4e17762523f1f28a33c973 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/140fd016-5c47-4166-97d2-40d8b9b6c56c_isrForCrosstalkSources_946479_6.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/140fd016-5c47-4166-97d2-40d8b9b6c56c_isrForCrosstalkSources_946479_6.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 747a6f393b9bb7f762ded338d1f88eee with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a240fcdb-d08e-40e3-a52f-763dcfde4164_isrForCrosstalkSources_946478_7.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a240fcdb-d08e-40e3-a52f-763dcfde4164_isrForCrosstalkSources_946478_7.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 420a9ea859cf9f9cb2a24425a2a483ac with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/fd77d0b3-f598-4f2d-b694-a6487f578f1e_isrForCrosstalkSources_946478_40.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/fd77d0b3-f598-4f2d-b694-a6487f578f1e_isrForCrosstalkSources_946478_40.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31ace38ee05105f0239101a3702625cc with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/2c6ba1c2-fef4-4136-bc8d-7c83e25766e2_isrForCrosstalkSources_946478_55.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/2c6ba1c2-fef4-4136-bc8d-7c83e25766e2_isrForCrosstalkSources_946478_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:51 proc_lsst.multi:146 [INFO] found job 17025261 in provider escience_a40 proc_lsst.multi INFO: found job 17025261 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:51 proc_lsst.multi:146 [INFO] found job 17025262 in provider escience_a40 proc_lsst.multi INFO: found job 17025262 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:52 proc_lsst.multi:146 [INFO] found job 17025263 in provider escience_a40 proc_lsst.multi INFO: found job 17025263 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:52 proc_lsst.multi:146 [INFO] found job 17025264 in provider escience_a40 proc_lsst.multi INFO: found job 17025264 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:52 proc_lsst.multi:146 [INFO] found job 17025265 in provider escience_a40 proc_lsst.multi INFO: found job 17025265 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6dce8f1e02c66e8f2acf38e98b3ff56 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c4086778-c8ac-420c-9f98-ae24054808bc_isrForCrosstalkSources_946369_32.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c4086778-c8ac-420c-9f98-ae24054808bc_isrForCrosstalkSources_946369_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:52 proc_lsst.multi:146 [INFO] found job 17025266 in provider escience_a40 proc_lsst.multi INFO: found job 17025266 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fafb746b7a7f973ca36afd4ceb3c451b with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/2dc73e27-8136-42af-810b-d3e5390eb4c7_isrForCrosstalkSources_946479_39.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/2dc73e27-8136-42af-810b-d3e5390eb4c7_isrForCrosstalkSources_946479_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:53 proc_lsst.multi:146 [INFO] found job 17025267 in provider escience_a40 proc_lsst.multi INFO: found job 17025267 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b480857b3b56ac9cfaa3d78bb8c389c7 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/02efcb4f-7142-4832-bd6d-7b304b4e4fce_isrForCrosstalkSources_946369_19.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/02efcb4f-7142-4832-bd6d-7b304b4e4fce_isrForCrosstalkSources_946369_19.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49f978599e483e36863da311b1f03fbb with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/581ad702-ad4e-4783-8aac-ef1322502524_isrForCrosstalkSources_946370_22.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/581ad702-ad4e-4783-8aac-ef1322502524_isrForCrosstalkSources_946370_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:53 proc_lsst.multi:146 [INFO] found job 17025268 in provider escience_a40 proc_lsst.multi INFO: found job 17025268 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 880d308d885deb633478d46d3c9f3acb with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ca2d5076-0492-4050-a68d-e5c61d2356aa_isrForCrosstalkSources_946478_1.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ca2d5076-0492-4050-a68d-e5c61d2356aa_isrForCrosstalkSources_946478_1.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea3b6bc7adc300dc80f3643f41588bdf with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6fec7f49-e661-4a0a-ae9b-017f11b580f4_isrForCrosstalkSources_946478_44.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6fec7f49-e661-4a0a-ae9b-017f11b580f4_isrForCrosstalkSources_946478_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:53 proc_lsst.multi:146 [INFO] found job 17025269 in provider escience_a40 proc_lsst.multi INFO: found job 17025269 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43f1ac2f1994ea29976e8e00e70fc6c8 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/9f60ea8a-88e3-4212-ba00-2b13ed1400a7_isrForCrosstalkSources_946371_43.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/9f60ea8a-88e3-4212-ba00-2b13ed1400a7_isrForCrosstalkSources_946371_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:54 proc_lsst.multi:146 [INFO] found job 17025270 in provider escience_a40 proc_lsst.multi INFO: found job 17025270 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a24f636e02aa931544a426a7a29f28e with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3770f4bd-8827-4106-b70f-821a541f7264_isrForCrosstalkSources_946479_49.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3770f4bd-8827-4106-b70f-821a541f7264_isrForCrosstalkSources_946479_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:54 proc_lsst.multi:146 [INFO] found job 17025271 in provider escience_a40 proc_lsst.multi INFO: found job 17025271 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bd06f5b052a71f1f753b1813787f780 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/64dd8b81-158d-4972-8e44-38e4f74bedba_isrForCrosstalkSources_946370_52.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/64dd8b81-158d-4972-8e44-38e4f74bedba_isrForCrosstalkSources_946370_52.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca04db9837f7fc108600e7b499fc358e with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e445590e-cbb9-4c35-95d5-3e408106efee_isrForCrosstalkSources_946479_40.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e445590e-cbb9-4c35-95d5-3e408106efee_isrForCrosstalkSources_946479_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:54 proc_lsst.multi:146 [INFO] found job 17025272 in provider escience_a40 proc_lsst.multi INFO: found job 17025272 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:54 proc_lsst.multi:146 [INFO] found job 17025273 in provider escience_a40 proc_lsst.multi INFO: found job 17025273 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5efa6e49e064f6b74a2f95193e17ef17 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/d10daa43-88c4-464d-b0e9-20b63ea1072f_isrForCrosstalkSources_946477_32.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/d10daa43-88c4-464d-b0e9-20b63ea1072f_isrForCrosstalkSources_946477_32.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b05e38d5b959d1d9b7660eb26d3644c3 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5083b6f5-49dc-4315-af83-db9f567ce092_isrForCrosstalkSources_946369_44.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5083b6f5-49dc-4315-af83-db9f567ce092_isrForCrosstalkSources_946369_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:54 proc_lsst.multi:146 [INFO] found job 17025274 in provider escience_a40 proc_lsst.multi INFO: found job 17025274 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f80452cc60aa9a7bd1b625191b791ae with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d7aae3c7-d542-46c7-9a9c-3badb320934d_isrForCrosstalkSources_946478_13.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d7aae3c7-d542-46c7-9a9c-3badb320934d_isrForCrosstalkSources_946478_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:54 proc_lsst.multi:146 [INFO] found job 17025275 in provider escience_a40 proc_lsst.multi INFO: found job 17025275 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f26caf46ad030e1e1444fe8f64ff2f3 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/4e40be26-0305-4e78-89d8-6bba1f5d4df1_isrForCrosstalkSources_946479_37.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/4e40be26-0305-4e78-89d8-6bba1f5d4df1_isrForCrosstalkSources_946479_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:55 proc_lsst.multi:146 [INFO] found job 17025276 in provider escience_a40 proc_lsst.multi INFO: found job 17025276 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0dc71d2978d3b67cf3e50b2b0e3cd67 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d6f5e1f2-4c77-43d1-9209-2279395d4ae2_isrForCrosstalkSources_946478_16.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d6f5e1f2-4c77-43d1-9209-2279395d4ae2_isrForCrosstalkSources_946478_16.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cdbad83151fa41cc048ede251a989f4 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/8e9ba0f5-a32e-457b-bfe6-f465a9cd5dbb_isrForCrosstalkSources_946477_28.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/8e9ba0f5-a32e-457b-bfe6-f465a9cd5dbb_isrForCrosstalkSources_946477_28.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 820042b7cb7487c07df30aa400cd4071 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/e938c4de-8308-40f2-a9a4-7833d5d451f3_isrForCrosstalkSources_946369_40.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/e938c4de-8308-40f2-a9a4-7833d5d451f3_isrForCrosstalkSources_946369_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:55 proc_lsst.multi:146 [INFO] found job 17025277 in provider escience_a40 proc_lsst.multi INFO: found job 17025277 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:55 proc_lsst.multi:146 [INFO] found job 17025278 in provider escience_a40 proc_lsst.multi INFO: found job 17025278 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:55 proc_lsst.multi:146 [INFO] found job 17025279 in provider escience_a40 proc_lsst.multi INFO: found job 17025279 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49f9c26405d3dfdc050b01eee2150618 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/bd5c48c2-e2d9-41f9-90ff-044752d7e9bd_isrForCrosstalkSources_946370_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/bd5c48c2-e2d9-41f9-90ff-044752d7e9bd_isrForCrosstalkSources_946370_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:55 proc_lsst.multi:146 [INFO] found job 17025280 in provider escience_a40 proc_lsst.multi INFO: found job 17025280 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:56 proc_lsst.multi:146 [INFO] found job 17025281 in provider checkpoint proc_lsst.multi INFO: found job 17025281 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebf0a6e5466791d8bba0407ba0d98475 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/dbdfeca6-4c80-487a-a184-1c9fa7c2357d_isrForCrosstalkSources_946477_40.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/dbdfeca6-4c80-487a-a184-1c9fa7c2357d_isrForCrosstalkSources_946477_40.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac20067b62d6498947755ee40aef6d39 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3249015c-ec8b-42e4-874c-1ea3223c1a39_isrForCrosstalkSources_946479_61.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3249015c-ec8b-42e4-874c-1ea3223c1a39_isrForCrosstalkSources_946479_61.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb8ec27756ea49088796e93e89df07dd with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a487eac3-a072-41c3-ab46-8d0777c4a33d_isrForCrosstalkSources_946478_54.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a487eac3-a072-41c3-ab46-8d0777c4a33d_isrForCrosstalkSources_946478_54.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 402bba84391b7645418fcec4538b3c6d with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6cc8da6c-d332-40b9-983e-bf3751f07b92_isrForCrosstalkSources_946478_37.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6cc8da6c-d332-40b9-983e-bf3751f07b92_isrForCrosstalkSources_946478_37.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ee4b063c0b6a05c553ad89e4ad99a23 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/463ce0ca-82ed-44eb-ad34-3bebac9a3213_isrForCrosstalkSources_946370_6.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/463ce0ca-82ed-44eb-ad34-3bebac9a3213_isrForCrosstalkSources_946370_6.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c4b2e5db68250e41b6c79a1cb95a55d with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2bcd9380-9546-4335-b254-808f4ba547dc_isrForCrosstalkSources_946477_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2bcd9380-9546-4335-b254-808f4ba547dc_isrForCrosstalkSources_946477_2.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51a5fd7cf987ec3ab0f63fc8226149a1 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/7d37e929-aeed-4083-a3d9-7be660911548_isrForCrosstalkSources_946370_36.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/7d37e929-aeed-4083-a3d9-7be660911548_isrForCrosstalkSources_946370_36.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae37e3cea420b04e07abf4c020cd3855 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5c8a4f16-d11d-4c34-b253-f1282bce87ca_isrForCrosstalkSources_946369_22.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5c8a4f16-d11d-4c34-b253-f1282bce87ca_isrForCrosstalkSources_946369_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:56 proc_lsst.multi:146 [INFO] found job 17025282 in provider checkpoint proc_lsst.multi INFO: found job 17025282 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:56 proc_lsst.multi:146 [INFO] found job 17025283 in provider checkpoint proc_lsst.multi INFO: found job 17025283 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:57 proc_lsst.multi:146 [INFO] found job 17025284 in provider checkpoint proc_lsst.multi INFO: found job 17025284 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:57 proc_lsst.multi:146 [INFO] found job 17025285 in provider checkpoint proc_lsst.multi INFO: found job 17025285 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddc0bcf7a2983204a374215592f7d5a2 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/48951240-979b-4d58-adde-22b069f1f603_isrForCrosstalkSources_946370_4.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/48951240-979b-4d58-adde-22b069f1f603_isrForCrosstalkSources_946370_4.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6e397cd9d399d45577310a14d807ea0 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6e17cf75-988e-4c72-98e7-d08c877db52b_isrForCrosstalkSources_946477_6.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6e17cf75-988e-4c72-98e7-d08c877db52b_isrForCrosstalkSources_946477_6.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4adff40ec398dd426cd45d807d4f4fa with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/99d3d3be-2769-4db4-9a24-f007d7c0041d_isrForCrosstalkSources_946369_62.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/99d3d3be-2769-4db4-9a24-f007d7c0041d_isrForCrosstalkSources_946369_62.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39be7dbd947b3371222e4c9d987e7cdd with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3f2bf86e-0ab7-443c-bd98-9882573497f8_isrForCrosstalkSources_946477_20.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3f2bf86e-0ab7-443c-bd98-9882573497f8_isrForCrosstalkSources_946477_20.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72b755c148a429bfcf5b2bab1befaa82 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/f7209d4a-ac40-4a06-b990-702a9d4c616b_isrForCrosstalkSources_946369_21.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/f7209d4a-ac40-4a06-b990-702a9d4c616b_isrForCrosstalkSources_946369_21.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdbe0fc7f3dbff54d9246e5ce5b406fb with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f8479c10-7c84-41d9-a2dd-ef8b145f72f9_isrForCrosstalkSources_946478_36.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f8479c10-7c84-41d9-a2dd-ef8b145f72f9_isrForCrosstalkSources_946478_36.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2839f664bf11034777bb4f01bbbc42f2 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1858c8c9-a413-459e-b019-38ef2969dcf9_isrForCrosstalkSources_946370_25.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1858c8c9-a413-459e-b019-38ef2969dcf9_isrForCrosstalkSources_946370_25.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 175b85b368b661fd782ab401f0311ac3 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ca5eb188-af0b-47de-b9d0-8a714968aacd_isrForCrosstalkSources_946478_6.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ca5eb188-af0b-47de-b9d0-8a714968aacd_isrForCrosstalkSources_946478_6.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 957808e4c91d18dfb2752e5a129cbe35 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/3c39584d-9233-4577-a582-e42061b33633_isrForCrosstalkSources_946370_46.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/3c39584d-9233-4577-a582-e42061b33633_isrForCrosstalkSources_946370_46.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d29b5220548d310d60e941b3bca671e5 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/93de60b2-ff80-4b9f-83fd-1e6f95fc94d4_isrForCrosstalkSources_946479_30.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/93de60b2-ff80-4b9f-83fd-1e6f95fc94d4_isrForCrosstalkSources_946479_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:57 proc_lsst.multi:146 [INFO] found job 17025286 in provider checkpoint proc_lsst.multi INFO: found job 17025286 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:57 proc_lsst.multi:146 [INFO] found job 17025287 in provider checkpoint proc_lsst.multi INFO: found job 17025287 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:57 proc_lsst.multi:146 [INFO] found job 17025288 in provider checkpoint proc_lsst.multi INFO: found job 17025288 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef9c9a1b914cd0b8a6689dbbe3712f96 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/25e69276-bfd6-4bf8-a095-207e00809919_isrForCrosstalkSources_946479_44.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/25e69276-bfd6-4bf8-a095-207e00809919_isrForCrosstalkSources_946479_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:57 proc_lsst.multi:146 [INFO] found job 17025289 in provider checkpoint proc_lsst.multi INFO: found job 17025289 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:58 proc_lsst.multi:146 [INFO] found job 17025290 in provider checkpoint proc_lsst.multi INFO: found job 17025290 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12b1dab222aac7ebe05de6cd4e8c5e42 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/5bde6491-37d1-46f6-9cd7-594c80723114_isrForCrosstalkSources_946477_58.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/5bde6491-37d1-46f6-9cd7-594c80723114_isrForCrosstalkSources_946477_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:58 proc_lsst.multi:146 [INFO] found job 17025291 in provider checkpoint proc_lsst.multi INFO: found job 17025291 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a062f3537dc3d34c8eb54033bdf0f95 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/813a6654-cb8b-41b3-9d45-d96363f1ccd8_isrForCrosstalkSources_946369_34.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/813a6654-cb8b-41b3-9d45-d96363f1ccd8_isrForCrosstalkSources_946369_34.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4666fa49c68d898917f244612bf83948 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c1ab57d2-6192-48f6-a4e0-035da39292c3_isrForCrosstalkSources_946369_46.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c1ab57d2-6192-48f6-a4e0-035da39292c3_isrForCrosstalkSources_946369_46.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fba5923148bcff2f276ea46f177e2003 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/10910ece-f063-4074-b65d-839112b0e6e0_isrForCrosstalkSources_946369_39.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/10910ece-f063-4074-b65d-839112b0e6e0_isrForCrosstalkSources_946369_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:58 proc_lsst.multi:146 [INFO] found job 17025292 in provider checkpoint proc_lsst.multi INFO: found job 17025292 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 032d86527086254bd86280f1979fe720 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3a0c5383-af37-45c2-b336-88edca89a472_isrForCrosstalkSources_946479_19.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3a0c5383-af37-45c2-b336-88edca89a472_isrForCrosstalkSources_946479_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:58 proc_lsst.multi:146 [INFO] found job 17025293 in provider checkpoint proc_lsst.multi INFO: found job 17025293 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76961ad00b65d981c56306ac0bc652e2 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1b68e782-fc79-43d6-acd8-319c5005da28_isrForCrosstalkSources_946369_58.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1b68e782-fc79-43d6-acd8-319c5005da28_isrForCrosstalkSources_946369_58.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb3c2246c492aa85ba435fbdaea94d37 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0bc9f122-5922-44f9-bcaa-51c5ad62d2b4_isrForCrosstalkSources_946371_56.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0bc9f122-5922-44f9-bcaa-51c5ad62d2b4_isrForCrosstalkSources_946371_56.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaa7f082e8b3b92f04e31b318e323af9 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/dd9b1132-70cc-4ef3-8bec-5cf0b64a4ac8_isrForCrosstalkSources_946478_56.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/dd9b1132-70cc-4ef3-8bec-5cf0b64a4ac8_isrForCrosstalkSources_946478_56.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 790f03a14d5e0278794c452d039d32a3 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/861b1a92-9631-4d32-8874-106401a8c0d4_isrForCrosstalkSources_946478_21.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/861b1a92-9631-4d32-8874-106401a8c0d4_isrForCrosstalkSources_946478_21.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47f98d7bbf38e02b643d446f2b3278f4 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f309b70b-f7e9-4d5f-b722-aa4abba5e74b_isrForCrosstalkSources_946478_51.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f309b70b-f7e9-4d5f-b722-aa4abba5e74b_isrForCrosstalkSources_946478_51.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6acda13309f184fb68a206c3c43f5438 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4681e298-f395-466a-8af1-d87584e32717_isrForCrosstalkSources_946371_10.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4681e298-f395-466a-8af1-d87584e32717_isrForCrosstalkSources_946371_10.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6abcf071d76ecc717182247ff4e871e with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/98afbdc1-e1db-465e-89aa-254758112f83_isrForCrosstalkSources_946370_23.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/98afbdc1-e1db-465e-89aa-254758112f83_isrForCrosstalkSources_946370_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:58 proc_lsst.multi:146 [INFO] found job 17025294 in provider checkpoint proc_lsst.multi INFO: found job 17025294 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42c2169671b3c4c695c002fae1aca791 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/6df1d4f3-c789-441a-b8f5-fc0f316651b3_isrForCrosstalkSources_946369_1.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/6df1d4f3-c789-441a-b8f5-fc0f316651b3_isrForCrosstalkSources_946369_1.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a24c9c38cde060f11fb9e4f2878ca54 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0bb6ff00-a6b3-4e3b-af9a-992c791138b9_isrForCrosstalkSources_946369_7.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0bb6ff00-a6b3-4e3b-af9a-992c791138b9_isrForCrosstalkSources_946369_7.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67d970b7855b6dc66c1d68701badf1b1 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a0a530f3-56ed-433d-bdd3-1cc62357685f_isrForCrosstalkSources_946478_41.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a0a530f3-56ed-433d-bdd3-1cc62357685f_isrForCrosstalkSources_946478_41.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 577e5a11ad385f6d5cd1e6b10d7db8b5 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e5adc0e7-8fd5-4a6c-8aba-4454d426dee6_isrForCrosstalkSources_946477_9.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e5adc0e7-8fd5-4a6c-8aba-4454d426dee6_isrForCrosstalkSources_946477_9.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bb8db2f3d7166c19c8bcd10dca10f00 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4442901d-17ac-427a-95af-88ada76f36c2_isrForCrosstalkSources_946477_15.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4442901d-17ac-427a-95af-88ada76f36c2_isrForCrosstalkSources_946477_15.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef8c598847550b8cd0356b95e49e5242 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/f9fb522f-4977-40fd-a692-d51c9c851232_isrForCrosstalkSources_946370_3.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/f9fb522f-4977-40fd-a692-d51c9c851232_isrForCrosstalkSources_946370_3.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 384ca7d7b0c8d4ad9fcd0b63d5ee42a8 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/643875a3-e0fb-4c1d-9bc7-8d16ce864155_isrForCrosstalkSources_946477_41.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/643875a3-e0fb-4c1d-9bc7-8d16ce864155_isrForCrosstalkSources_946477_41.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e179ea2f4c0c3beac4bedd18165ec6a with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/8d8a8c58-d2c9-4a94-9ec7-40dee328b5a2_isrForCrosstalkSources_946479_43.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/8d8a8c58-d2c9-4a94-9ec7-40dee328b5a2_isrForCrosstalkSources_946479_43.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 894571f43fa42ea11eff71ce4aa6a312 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/c32fbb8b-fce2-4b90-8134-29fe27fe8d9b_isrForCrosstalkSources_946477_62.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/c32fbb8b-fce2-4b90-8134-29fe27fe8d9b_isrForCrosstalkSources_946477_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:59 proc_lsst.multi:146 [INFO] found job 17025295 in provider checkpoint proc_lsst.multi INFO: found job 17025295 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b291246eba27aeef0373a03484a069a with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2f248384-9980-4cad-9e82-79c5bcbf950a_isrForCrosstalkSources_946369_37.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2f248384-9980-4cad-9e82-79c5bcbf950a_isrForCrosstalkSources_946369_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:59 proc_lsst.multi:146 [INFO] found job 17025296 in provider checkpoint proc_lsst.multi INFO: found job 17025296 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:13:59 proc_lsst.multi:146 [INFO] found job 17025297 in provider checkpoint proc_lsst.multi INFO: found job 17025297 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:00 proc_lsst.multi:146 [INFO] found job 17025298 in provider checkpoint proc_lsst.multi INFO: found job 17025298 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:00 proc_lsst.multi:146 [INFO] found job 17025299 in provider checkpoint proc_lsst.multi INFO: found job 17025299 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:00 proc_lsst.multi:146 [INFO] found job 17025300 in provider checkpoint proc_lsst.multi INFO: found job 17025300 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a73d1caacdfdb0f8025e643c9a718a1f with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/630cee2a-48f4-41a7-bd82-7b2d4880a3b6_isrForCrosstalkSources_946479_14.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/630cee2a-48f4-41a7-bd82-7b2d4880a3b6_isrForCrosstalkSources_946479_14.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e74d9ca40f036421ed538540f028fde9 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/d7f68d1b-a690-4a47-86a1-2a9610397ead_isrForCrosstalkSources_946370_48.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/d7f68d1b-a690-4a47-86a1-2a9610397ead_isrForCrosstalkSources_946370_48.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afd8e5e033a6f0977f87ff304fbeb2ca with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/1980e801-91fd-4750-ba4b-0622b785c66b_isrForCrosstalkSources_946479_2.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/1980e801-91fd-4750-ba4b-0622b785c66b_isrForCrosstalkSources_946479_2.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a4f1954e69a1b02377a7cded7f43fac with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d73ceabc-ae6c-4f7e-846e-33f1f061dcc1_isrForCrosstalkSources_946371_7.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d73ceabc-ae6c-4f7e-846e-33f1f061dcc1_isrForCrosstalkSources_946371_7.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9a0b8d9269843602a610ce8d7339a57 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/93b8ff06-bcc3-461c-832f-dfe268aa17e7_isrForCrosstalkSources_946479_26.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/93b8ff06-bcc3-461c-832f-dfe268aa17e7_isrForCrosstalkSources_946479_26.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8693f976257a5be257e611089eca811 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d3c63323-b9af-4edb-b88b-4e8f28f441db_isrForCrosstalkSources_946478_59.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d3c63323-b9af-4edb-b88b-4e8f28f441db_isrForCrosstalkSources_946478_59.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49fba05e92b764a2eeac6c290d4fbb89 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0da1a2a4-c1f0-4484-8442-dce2befdb4d0_isrForCrosstalkSources_946477_48.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0da1a2a4-c1f0-4484-8442-dce2befdb4d0_isrForCrosstalkSources_946477_48.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c023265a44238f3f9b7661120419c54c with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/a7193dc7-3a73-4d2a-83e3-9cf11117f78b_isrForCrosstalkSources_946370_20.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/a7193dc7-3a73-4d2a-83e3-9cf11117f78b_isrForCrosstalkSources_946370_20.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 606e488a3062f8bdab58214c73609677 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/9378bef4-6902-431f-a077-2b84da694ff8_isrForCrosstalkSources_946370_61.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/9378bef4-6902-431f-a077-2b84da694ff8_isrForCrosstalkSources_946370_61.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 170c89feb552bd6a956917fbd7995af8 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/498bffc5-976d-40a2-825a-b7b169e04bf0_isrForCrosstalkSources_946370_50.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/498bffc5-976d-40a2-825a-b7b169e04bf0_isrForCrosstalkSources_946370_50.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83ad4449d698df076f853f84dca456e8 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e181bf53-fe38-402f-a243-adabba0db748_isrForCrosstalkSources_946478_35.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e181bf53-fe38-402f-a243-adabba0db748_isrForCrosstalkSources_946478_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:01 proc_lsst.multi:146 [INFO] found job 17025301 in provider checkpoint proc_lsst.multi INFO: found job 17025301 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:01 proc_lsst.multi:146 [INFO] found job 17025302 in provider checkpoint proc_lsst.multi INFO: found job 17025302 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:01 proc_lsst.multi:146 [INFO] found job 17025303 in provider checkpoint proc_lsst.multi INFO: found job 17025303 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96d03ee87fd7ae750a80cff61585732e with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/8d10fa7b-32a9-49c7-83d2-c35b1fc3eced_isrForCrosstalkSources_946478_60.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/8d10fa7b-32a9-49c7-83d2-c35b1fc3eced_isrForCrosstalkSources_946478_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:01 proc_lsst.multi:146 [INFO] found job 17025304 in provider checkpoint proc_lsst.multi INFO: found job 17025304 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f2a8ca5180c396befe7dd933b1b00b9 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/88417e92-eedc-4da4-9b6d-0c5edf534e29_isrForCrosstalkSources_946370_51.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/88417e92-eedc-4da4-9b6d-0c5edf534e29_isrForCrosstalkSources_946370_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:02 proc_lsst.multi:146 [INFO] found job 17025305 in provider checkpoint proc_lsst.multi INFO: found job 17025305 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:02 proc_lsst.multi:146 [INFO] found job 17025306 in provider checkpoint proc_lsst.multi INFO: found job 17025306 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 160fad007f2486cf1fe59b3c72436fcb with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4c1054e8-9c75-4a03-a672-a8df3f33fb42_isrForCrosstalkSources_946369_56.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4c1054e8-9c75-4a03-a672-a8df3f33fb42_isrForCrosstalkSources_946369_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:02 proc_lsst.multi:146 [INFO] found job 17025307 in provider checkpoint proc_lsst.multi INFO: found job 17025307 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:02 proc_lsst.multi:146 [INFO] found job 17025308 in provider checkpoint proc_lsst.multi INFO: found job 17025308 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeb24e914e9d53dfb538ea007fa8b154 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6f746e01-a308-4dd6-bf6b-40cb7bd79a71_isrForCrosstalkSources_946371_59.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6f746e01-a308-4dd6-bf6b-40cb7bd79a71_isrForCrosstalkSources_946371_59.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe724735a722c5d797cb3e0fb7420e33 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/aea4fc84-f72a-40dd-9477-a5ce5b27f22b_isrForCrosstalkSources_946371_62.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/aea4fc84-f72a-40dd-9477-a5ce5b27f22b_isrForCrosstalkSources_946371_62.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2876f37011a4b40875eafee067e4753 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/50982bbe-5b4e-4269-8ffb-aa94cef7cde1_isrForCrosstalkSources_946479_45.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/50982bbe-5b4e-4269-8ffb-aa94cef7cde1_isrForCrosstalkSources_946479_45.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df20b6b11a33b6bf843a2e80230515cb with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0923c487-13d3-4649-a281-dd526341aef5_isrForCrosstalkSources_946477_33.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0923c487-13d3-4649-a281-dd526341aef5_isrForCrosstalkSources_946477_33.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24050ed132b6a5f69148caea947dc81b with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d387e504-00fb-4d9a-8f55-8b75f30d1428_isrForCrosstalkSources_946371_15.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/d387e504-00fb-4d9a-8f55-8b75f30d1428_isrForCrosstalkSources_946371_15.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ac7923a382d808ae3e3cb970dba78a7 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4124479d-fd9e-4219-8339-15425ad5b04f_isrForCrosstalkSources_946369_50.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4124479d-fd9e-4219-8339-15425ad5b04f_isrForCrosstalkSources_946369_50.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1102e814215cb10d2a80e2cc1ccc755e with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c83fcc71-7982-4a43-a6c6-0f9ed6ad26e1_isrForCrosstalkSources_946479_25.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c83fcc71-7982-4a43-a6c6-0f9ed6ad26e1_isrForCrosstalkSources_946479_25.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 273de1c2214e0a81063ee8843156cc17 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/134c748c-e33b-4f93-8b81-407c8861cc92_isrForCrosstalkSources_946478_12.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/134c748c-e33b-4f93-8b81-407c8861cc92_isrForCrosstalkSources_946478_12.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da55404abaa894f96d7e6f1339fc8f0b with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0ac4dd39-8e5f-44f0-8317-c33b5b2849a8_isrForCrosstalkSources_946370_57.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0ac4dd39-8e5f-44f0-8317-c33b5b2849a8_isrForCrosstalkSources_946370_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:02 proc_lsst.multi:146 [INFO] found job 17025309 in provider checkpoint proc_lsst.multi INFO: found job 17025309 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:02 proc_lsst.multi:146 [INFO] found job 17025310 in provider checkpoint proc_lsst.multi INFO: found job 17025310 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10470616df40f84263fe8597be53d9e5 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/389ca738-025b-47fd-a66c-d58dddd25b48_isrForCrosstalkSources_946479_34.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/389ca738-025b-47fd-a66c-d58dddd25b48_isrForCrosstalkSources_946479_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:02 proc_lsst.multi:146 [INFO] found job 17025311 in provider checkpoint proc_lsst.multi INFO: found job 17025311 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccecf045f1b826d9c5a0471fd4f6a51b with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c00ccc8f-2079-4000-bc07-6c05e1e5435d_isrForCrosstalkSources_946479_27.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c00ccc8f-2079-4000-bc07-6c05e1e5435d_isrForCrosstalkSources_946479_27.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1f1d8efeaf159ea1bf6f403a3f6676 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/34f7c2ed-1712-41e2-bb98-c3aa3862cc1d_isrForCrosstalkSources_946371_14.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/34f7c2ed-1712-41e2-bb98-c3aa3862cc1d_isrForCrosstalkSources_946371_14.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb6ecae4e68437dabd19a4cc875d39c2 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a2aee55e-433f-48a6-85b8-e4a4d9603ddb_isrForCrosstalkSources_946371_8.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a2aee55e-433f-48a6-85b8-e4a4d9603ddb_isrForCrosstalkSources_946371_8.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aceef3f578691f6f7565f9cc738b2e07 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e1c91ed3-28b2-471d-957d-d6e8a1fd5a54_isrForCrosstalkSources_946479_5.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e1c91ed3-28b2-471d-957d-d6e8a1fd5a54_isrForCrosstalkSources_946479_5.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7af6a6b22e0831da87db9c33200a0125 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/510c5daf-5017-4791-9847-3ace8ee1eee7_isrForCrosstalkSources_946479_7.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/510c5daf-5017-4791-9847-3ace8ee1eee7_isrForCrosstalkSources_946479_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:03 proc_lsst.multi:146 [INFO] found job 17025312 in provider checkpoint proc_lsst.multi INFO: found job 17025312 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f895f1e5bd210896a3c92446293bb160 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/ec9ce60b-1c62-446a-a6b4-762cf601900a_isrForCrosstalkSources_946479_16.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/ec9ce60b-1c62-446a-a6b4-762cf601900a_isrForCrosstalkSources_946479_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:03 proc_lsst.multi:146 [INFO] found job 17025313 in provider checkpoint proc_lsst.multi INFO: found job 17025313 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc1216159a55500b0bb83add19c359a2 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e33397d8-883d-40d2-87f9-560aa987a36b_isrForCrosstalkSources_946479_56.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e33397d8-883d-40d2-87f9-560aa987a36b_isrForCrosstalkSources_946479_56.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4bd597705d2ff2b796f10ad3bcb703 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/826f55fd-13ba-48f6-a54a-c07c76622a75_isrForCrosstalkSources_946478_23.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/826f55fd-13ba-48f6-a54a-c07c76622a75_isrForCrosstalkSources_946478_23.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36bc67a6033c5cb5643155cd006e5acd with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/988d9c5c-0270-49c2-a16f-230608bf1405_isrForCrosstalkSources_946371_40.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/988d9c5c-0270-49c2-a16f-230608bf1405_isrForCrosstalkSources_946371_40.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b49af9e1b14894f81da8eb464069873 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/62b2379d-c56f-443f-9e58-e1b548864015_isrForCrosstalkSources_946369_45.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/62b2379d-c56f-443f-9e58-e1b548864015_isrForCrosstalkSources_946369_45.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83703b286350bb472b5fe52715afe58f with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/84e890e2-06fb-4c33-9b96-c3d232438871_isrForCrosstalkSources_946370_14.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/84e890e2-06fb-4c33-9b96-c3d232438871_isrForCrosstalkSources_946370_14.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daa6a13e6cdb4c21014930f26f14f9eb with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/8edb4d46-77fe-4847-a7a9-a97be9875fe8_isrForCrosstalkSources_946370_8.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/8edb4d46-77fe-4847-a7a9-a97be9875fe8_isrForCrosstalkSources_946370_8.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a2c434efada220b5c1e3ca8e0cda509 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/a5515d24-605f-4349-9afc-0d8d75c757d7_isrForCrosstalkSources_946369_24.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/a5515d24-605f-4349-9afc-0d8d75c757d7_isrForCrosstalkSources_946369_24.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9eb2db7c8e8aa019504ea959d1e31d0 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/3ef9b635-c9f2-460d-827c-36d623edf9d1_isrForCrosstalkSources_946371_36.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/3ef9b635-c9f2-460d-827c-36d623edf9d1_isrForCrosstalkSources_946371_36.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e933fd874f27725ea3fe6f294528bcc with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/41b64316-42c3-4f7c-8d2b-c17be6f407b6_isrForCrosstalkSources_946478_3.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/41b64316-42c3-4f7c-8d2b-c17be6f407b6_isrForCrosstalkSources_946478_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:03 proc_lsst.multi:146 [INFO] found job 17025314 in provider checkpoint proc_lsst.multi INFO: found job 17025314 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7109631f4b0288684175e8cab56789c with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/54c3998c-d33a-4060-87f5-764117b0f80a_isrForCrosstalkSources_946479_11.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/54c3998c-d33a-4060-87f5-764117b0f80a_isrForCrosstalkSources_946479_11.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9738479699053f4c4d3095968a95b96 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e4dc26f1-2f36-47ce-bebf-199e32475cfb_isrForCrosstalkSources_946478_18.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e4dc26f1-2f36-47ce-bebf-199e32475cfb_isrForCrosstalkSources_946478_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:03 proc_lsst.multi:146 [INFO] found job 17025315 in provider checkpoint proc_lsst.multi INFO: found job 17025315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb7e1cff49da4f4bba2f3f246d842c21 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f41b50ee-0563-4574-b638-0226e94267e0_isrForCrosstalkSources_946477_25.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f41b50ee-0563-4574-b638-0226e94267e0_isrForCrosstalkSources_946477_25.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a5fdfc4a0e01258fe945265fdb1285d with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/827f0d3c-fbea-4d74-a7fe-915c23fc6f79_isrForCrosstalkSources_946479_62.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/827f0d3c-fbea-4d74-a7fe-915c23fc6f79_isrForCrosstalkSources_946479_62.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe72497e98b9debeea7e3394cc012e6 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/846f330a-0084-4429-8a27-b36eec8ea226_isrForCrosstalkSources_946477_30.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/846f330a-0084-4429-8a27-b36eec8ea226_isrForCrosstalkSources_946477_30.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 582ef90c00e298c670dd8e7a187f1d14 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1490802c-7152-4ca8-86ba-9f9a96de165b_isrForCrosstalkSources_946369_61.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1490802c-7152-4ca8-86ba-9f9a96de165b_isrForCrosstalkSources_946369_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:03 proc_lsst.multi:146 [INFO] found job 17025316 in provider checkpoint proc_lsst.multi INFO: found job 17025316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90f2432ff8a37332848c3e6faaf5e0a4 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/26460932-01be-439e-bb57-623c7a5806c7_isrForCrosstalkSources_946369_13.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/26460932-01be-439e-bb57-623c7a5806c7_isrForCrosstalkSources_946369_13.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29f3ef56ecaa10ce2dace6ecfa388ef8 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/31e7b1c9-aff8-4da9-a13b-424ca6e3ade9_isrForCrosstalkSources_946478_49.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/31e7b1c9-aff8-4da9-a13b-424ca6e3ade9_isrForCrosstalkSources_946478_49.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1af3dc83cbdaf9b400e36bb5cf097d70 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4a4eb0c0-87ab-4893-8525-f97474a5ced6_isrForCrosstalkSources_946477_21.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4a4eb0c0-87ab-4893-8525-f97474a5ced6_isrForCrosstalkSources_946477_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:03 proc_lsst.multi:146 [INFO] found job 17025317 in provider checkpoint proc_lsst.multi INFO: found job 17025317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d5e598b352bb54cafef436a5263fa98 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0eb25c5d-8ee0-47e0-ad45-db504847bcc7_isrForCrosstalkSources_946477_35.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0eb25c5d-8ee0-47e0-ad45-db504847bcc7_isrForCrosstalkSources_946477_35.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e108354e89720864c0a56d006ca5d25 with result from task 160 parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 225 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4b9e7deb-3ebd-4efe-a61a-33e24b20d326_isrForCrosstalkSources_946371_50.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4b9e7deb-3ebd-4efe-a61a-33e24b20d326_isrForCrosstalkSources_946371_50.stderr parsl.dataflow.strategy DEBUG: Executor multi has 225 active tasks, 84/9 running/pending blocks, and 177 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 225 parsl.dataflow.strategy DEBUG: Executor multi has 225 active tasks, 84/9 running/pending blocks, and 177 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edbdae557c82745d4ddbf97186667235 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c8115f97-e597-4ea9-a160-3a9e4436a5c5_isrForCrosstalkSources_946370_47.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c8115f97-e597-4ea9-a160-3a9e4436a5c5_isrForCrosstalkSources_946370_47.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c61160f6bf050d9471eb5d1ff529453 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/37c2d025-6933-49dd-b738-9e4311eb3c32_isrForCrosstalkSources_946371_41.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/37c2d025-6933-49dd-b738-9e4311eb3c32_isrForCrosstalkSources_946371_41.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13ec0cc5d262132b2b5bfb67a290d679 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/c3711d70-8b13-4a97-85ae-4e87b18c1637_isrForCrosstalkSources_946371_60.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/c3711d70-8b13-4a97-85ae-4e87b18c1637_isrForCrosstalkSources_946371_60.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97e1f806d04461631bf31571fd63a5d7 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/67a1605f-a864-4e06-884e-bab1a89c6368_isrForCrosstalkSources_946371_20.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/67a1605f-a864-4e06-884e-bab1a89c6368_isrForCrosstalkSources_946371_20.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ddd7cdccda6cdf799fcc649cb272f8b with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/5a1bebd6-4ff8-4609-adae-1c78e6254b27_isrForCrosstalkSources_946370_34.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/5a1bebd6-4ff8-4609-adae-1c78e6254b27_isrForCrosstalkSources_946370_34.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c7d02621c88b17307780cd65e5fa929 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4f6dacf5-c988-46bc-bbc2-b60df6b47576_isrForCrosstalkSources_946369_53.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/4f6dacf5-c988-46bc-bbc2-b60df6b47576_isrForCrosstalkSources_946369_53.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71e6118c76d178a6b7756f0d8b1f259a with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/fe5d1ca7-ae1c-429a-8501-38ef60c576de_isrForCrosstalkSources_946478_42.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/fe5d1ca7-ae1c-429a-8501-38ef60c576de_isrForCrosstalkSources_946478_42.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd4aca244edae083a9e647f8052e4abe with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/df8a6635-50a8-4f9e-b0f9-49a8636f26d7_isrForCrosstalkSources_946478_5.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/df8a6635-50a8-4f9e-b0f9-49a8636f26d7_isrForCrosstalkSources_946478_5.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81c1c68ad3b5f73034e65c0232de366e with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4a3e51fc-89b9-4118-a59a-5dfdd5e9b32c_isrForCrosstalkSources_946477_23.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/4a3e51fc-89b9-4118-a59a-5dfdd5e9b32c_isrForCrosstalkSources_946477_23.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d22a94c96e8eea542802302929aa5b80 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/29ed89d4-6b6d-4639-944f-fa4da6927220_isrForCrosstalkSources_946371_32.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/29ed89d4-6b6d-4639-944f-fa4da6927220_isrForCrosstalkSources_946371_32.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74abcdab785c8f67799b021aa900b44a with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/bcf0f4dc-4276-47ab-9ca8-d290748c2e4f_isrForCrosstalkSources_946370_5.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/bcf0f4dc-4276-47ab-9ca8-d290748c2e4f_isrForCrosstalkSources_946370_5.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7112fe3a6830e6e2ef689e893991e435 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/9853c604-9041-474c-8ca2-f890aff52176_isrForCrosstalkSources_946370_27.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/9853c604-9041-474c-8ca2-f890aff52176_isrForCrosstalkSources_946370_27.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66efcc3977d2658b9b8e31f7ca25ebba with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/6ea2e8e7-4909-4b0d-a665-20b383962b48_isrForCrosstalkSources_946369_28.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/6ea2e8e7-4909-4b0d-a665-20b383962b48_isrForCrosstalkSources_946369_28.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f7c4bf23ed0d3c90380e75fc1c4ef83 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/f1027f15-facc-4ea9-ac62-e4a224a18e50_isrForCrosstalkSources_946371_11.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/f1027f15-facc-4ea9-ac62-e4a224a18e50_isrForCrosstalkSources_946371_11.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 211 parsl.dataflow.strategy DEBUG: Executor multi has 211 active tasks, 84/9 running/pending blocks, and 281 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf69832902c374dd0660ce00cc492fcf with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/1839b9b2-ae84-4bb5-b9b7-2e96fae4ec66_isrForCrosstalkSources_946479_3.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/1839b9b2-ae84-4bb5-b9b7-2e96fae4ec66_isrForCrosstalkSources_946479_3.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f20c8c917b399ee79d5df6d7152a3ed with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0ec2425f-a5ee-429e-83cb-c36a4325bd0e_isrForCrosstalkSources_946371_16.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0ec2425f-a5ee-429e-83cb-c36a4325bd0e_isrForCrosstalkSources_946371_16.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 413e9c4f3ba0cce9250683f53d168f7d with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/89ec25c0-27d2-4210-8259-83f06774117c_isrForCrosstalkSources_946478_26.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/89ec25c0-27d2-4210-8259-83f06774117c_isrForCrosstalkSources_946478_26.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 496d409509b5438cfac0a67625e7bf62 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/fbe0627e-3298-41ce-9299-07eb41843371_isrForCrosstalkSources_946477_47.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/fbe0627e-3298-41ce-9299-07eb41843371_isrForCrosstalkSources_946477_47.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f46fea408a5f10506f302380d5e30d1 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/fd77f74c-1ac5-4d52-8782-631eb25cff46_isrForCrosstalkSources_946369_30.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/fd77f74c-1ac5-4d52-8782-631eb25cff46_isrForCrosstalkSources_946369_30.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d5d0d0aacdcf861f5c3e85f4ced51bc with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/ea84f01a-995a-4d75-93a1-d652b8948638_isrForCrosstalkSources_946369_31.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/ea84f01a-995a-4d75-93a1-d652b8948638_isrForCrosstalkSources_946369_31.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ac0322d9bd610d63b082f6a5de9a768 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c6d6a430-4273-4c75-b1d0-a18349b1de2e_isrForCrosstalkSources_946478_10.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c6d6a430-4273-4c75-b1d0-a18349b1de2e_isrForCrosstalkSources_946478_10.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac3758e8454d1466aa2e1c181a617d95 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6ad230a0-f9fb-4abe-aeac-6cbee9cdd410_isrForCrosstalkSources_946478_28.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6ad230a0-f9fb-4abe-aeac-6cbee9cdd410_isrForCrosstalkSources_946478_28.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b3dc36149fb3dbff6f7d30f991b6298 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/107684cd-8865-4426-94f0-3a89f1e0b3a4_isrForCrosstalkSources_946371_1.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/107684cd-8865-4426-94f0-3a89f1e0b3a4_isrForCrosstalkSources_946371_1.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9d5074b138193580c415fa6cb750074 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/96fd7ecd-855c-4241-91a7-1a5e97cdcaab_isrForCrosstalkSources_946370_24.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/96fd7ecd-855c-4241-91a7-1a5e97cdcaab_isrForCrosstalkSources_946370_24.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8d143289f50f4c7472520b594e58346 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/6a8e2571-b8dc-4dff-a870-ed3dcc7cd4cb_isrForCrosstalkSources_946370_9.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/6a8e2571-b8dc-4dff-a870-ed3dcc7cd4cb_isrForCrosstalkSources_946370_9.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 198423c6502331dcdef079be7246d888 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9d20ab43-126a-4582-96c3-c77a6fad106e_isrForCrosstalkSources_946477_26.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9d20ab43-126a-4582-96c3-c77a6fad106e_isrForCrosstalkSources_946477_26.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07a54a18d14549daacc6ef03991efd53 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3ec51067-66d1-4a49-8cba-9dde6ae16238_isrForCrosstalkSources_946477_13.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3ec51067-66d1-4a49-8cba-9dde6ae16238_isrForCrosstalkSources_946477_13.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8edfef2b9acdd7f4bdd188bd96d84ce0 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/7fb7ad48-921a-4e9a-800d-52b05ab82f1e_isrForCrosstalkSources_946477_14.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/7fb7ad48-921a-4e9a-800d-52b05ab82f1e_isrForCrosstalkSources_946477_14.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1682296de496dcad8d73b1c29323646 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/ef00a507-e04e-4df6-8280-963e5328cc88_isrForCrosstalkSources_946479_4.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/ef00a507-e04e-4df6-8280-963e5328cc88_isrForCrosstalkSources_946479_4.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56f3ad37d70f96652618e783bbe12c2e with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/40bf93f7-9323-45f6-a7ce-ed957f54849c_isrForCrosstalkSources_946369_20.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/40bf93f7-9323-45f6-a7ce-ed957f54849c_isrForCrosstalkSources_946369_20.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60fe37d9b0223dc5757736da52caf965 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e3b47d42-45c3-474e-b406-cc8df6b02168_isrForCrosstalkSources_946370_39.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e3b47d42-45c3-474e-b406-cc8df6b02168_isrForCrosstalkSources_946370_39.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8561022ae1b45414fbead8e86ec594f0 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/dc4af8c3-75bd-4798-81c6-f63800096751_isrForCrosstalkSources_946369_27.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/dc4af8c3-75bd-4798-81c6-f63800096751_isrForCrosstalkSources_946369_27.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 076700c5f16b4d1682b11ae7197359e7 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/a3ea80f3-a440-4b48-9f87-131b314acb8d_isrForCrosstalkSources_946370_15.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/a3ea80f3-a440-4b48-9f87-131b314acb8d_isrForCrosstalkSources_946370_15.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed4dca0dae8a8c7fe79df23cc0a41020 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/16c360e6-285d-436f-ac30-a3f45abf88b8_isrForCrosstalkSources_946479_22.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/16c360e6-285d-436f-ac30-a3f45abf88b8_isrForCrosstalkSources_946479_22.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78e382fc9442c89d96a047e72aa230e7 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/f139f80b-4665-43ef-80f6-5a968cef2ca3_isrForCrosstalkSources_946479_35.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/f139f80b-4665-43ef-80f6-5a968cef2ca3_isrForCrosstalkSources_946479_35.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f9959331a0fed2b14a48f09595a9180 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e54686dc-1c6d-4d4a-aff0-8bc387f26195_isrForCrosstalkSources_946370_16.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e54686dc-1c6d-4d4a-aff0-8bc387f26195_isrForCrosstalkSources_946370_16.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71dca5654755b23f4e48ca336f99cd9b with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/757e5924-070a-47c6-aa93-a0a7ed170613_isrForCrosstalkSources_946371_18.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/757e5924-070a-47c6-aa93-a0a7ed170613_isrForCrosstalkSources_946371_18.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b21968f6d67f8b096d42cd56f37d955 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/89f02670-8602-4c78-90cd-ed7d67ec8862_isrForCrosstalkSources_946477_17.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/89f02670-8602-4c78-90cd-ed7d67ec8862_isrForCrosstalkSources_946477_17.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1467e3692fbe65ca296dee16626a5bf with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4fd9ae36-a266-4ee5-ab60-e87f23712fbf_isrForCrosstalkSources_946370_54.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4fd9ae36-a266-4ee5-ab60-e87f23712fbf_isrForCrosstalkSources_946370_54.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f61968eb8e07f72466e45336ef278ab3 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/52b2f4f3-66cb-4317-9323-e15a1ce7fb4c_isrForCrosstalkSources_946479_24.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/52b2f4f3-66cb-4317-9323-e15a1ce7fb4c_isrForCrosstalkSources_946479_24.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3150e4eaa61c6340926f72daaf5a9d0f with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/bed16a6d-554a-440f-9f81-b22ad1654d75_isrForCrosstalkSources_946369_38.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/bed16a6d-554a-440f-9f81-b22ad1654d75_isrForCrosstalkSources_946369_38.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f13d34764d080cedefab30183d24c7d9 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/06df898f-9c34-4ee2-9ae8-04998a22ef01_isrForCrosstalkSources_946369_9.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/06df898f-9c34-4ee2-9ae8-04998a22ef01_isrForCrosstalkSources_946369_9.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40480f2f06b8187e24bd95038b0b0b2b with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/03903c5e-6d0c-424a-afe6-a921e44b93d0_isrForCrosstalkSources_946370_44.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/03903c5e-6d0c-424a-afe6-a921e44b93d0_isrForCrosstalkSources_946370_44.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8072820dc8c4f5807263072db9429019 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/b65a1809-d1ba-4607-8597-631b64315c22_isrForCrosstalkSources_946477_3.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/b65a1809-d1ba-4607-8597-631b64315c22_isrForCrosstalkSources_946477_3.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b14eae5ec4b62a45d1bfacf002895ba with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/19066285-7d7f-4ded-b9f6-76886416cd1b_isrForCrosstalkSources_946478_43.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/19066285-7d7f-4ded-b9f6-76886416cd1b_isrForCrosstalkSources_946478_43.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4ad39bec32722a3ff660f1f63041042 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/b9ff6af2-87da-45d5-88b0-c978977f05cb_isrForCrosstalkSources_946477_60.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/b9ff6af2-87da-45d5-88b0-c978977f05cb_isrForCrosstalkSources_946477_60.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0a650af85e75f786af67d7586368963 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/46320173-9709-4f00-a0af-9699284a6c89_isrForCrosstalkSources_946477_49.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/46320173-9709-4f00-a0af-9699284a6c89_isrForCrosstalkSources_946477_49.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e7250b3e70074acb07575f7fcf6a921 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/ed0c0a4d-7e93-4671-80eb-1be7e54cb0cc_isrForCrosstalkSources_946477_29.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/ed0c0a4d-7e93-4671-80eb-1be7e54cb0cc_isrForCrosstalkSources_946477_29.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b46da9ba76bb20b12d813a057202847 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/b3d60411-bce8-4893-87cb-ce49c011f1e8_isrForCrosstalkSources_946479_12.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/b3d60411-bce8-4893-87cb-ce49c011f1e8_isrForCrosstalkSources_946479_12.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2166a85129269ceb1ad8da082b05c97d with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/69c8e6c5-aba9-41d6-91b4-6fcbfc16982e_isrForCrosstalkSources_946477_59.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/69c8e6c5-aba9-41d6-91b4-6fcbfc16982e_isrForCrosstalkSources_946477_59.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9730dadc13ad1fa87e2291ef1de27985 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/ce14f43b-e2cd-42c3-a23a-18177457304b_isrForCrosstalkSources_946477_34.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/ce14f43b-e2cd-42c3-a23a-18177457304b_isrForCrosstalkSources_946477_34.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4442007222db8e4817dd399ae9440a3 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/c7128841-0228-44e4-a58b-78bceafb1d09_isrForCrosstalkSources_946371_49.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/c7128841-0228-44e4-a58b-78bceafb1d09_isrForCrosstalkSources_946371_49.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f9209851cfba3def5093f66793c536c with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/ce98cb28-f91c-498f-9026-aa43885b0266_isrForCrosstalkSources_946369_8.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/ce98cb28-f91c-498f-9026-aa43885b0266_isrForCrosstalkSources_946369_8.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8035b7afd674f35d694a28d067287a0 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/cf895075-8ffd-4f0c-89b0-cb2f2ed863af_isrForCrosstalkSources_946369_5.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/cf895075-8ffd-4f0c-89b0-cb2f2ed863af_isrForCrosstalkSources_946369_5.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f631dc93ae74fe5a3ce3145fefb29674 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/47877381-4542-4ff7-80a0-0989f9684f3b_isrForCrosstalkSources_946369_3.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/47877381-4542-4ff7-80a0-0989f9684f3b_isrForCrosstalkSources_946369_3.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf5e80f783dfd72c688843257771256 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/73326ac6-b5bd-46b7-8d55-96f23bcca694_isrForCrosstalkSources_946369_16.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/73326ac6-b5bd-46b7-8d55-96f23bcca694_isrForCrosstalkSources_946369_16.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07f090ebba1dd68f5aabf19f130a6f77 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/150d6b5c-a5ef-409e-aa05-6f57728f3790_isrForCrosstalkSources_946478_61.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/150d6b5c-a5ef-409e-aa05-6f57728f3790_isrForCrosstalkSources_946478_61.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58dc8f771977a67d62ec47ce777c905e with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/72b41ea4-0f7f-4b17-9994-c60e3ac70e42_isrForCrosstalkSources_946479_9.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/72b41ea4-0f7f-4b17-9994-c60e3ac70e42_isrForCrosstalkSources_946479_9.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7fb49fefc55bb58d5ebc4172cacdff with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0d7eb296-0944-4f0f-b081-f87e42ccabf6_isrForCrosstalkSources_946371_61.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0d7eb296-0944-4f0f-b081-f87e42ccabf6_isrForCrosstalkSources_946371_61.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afe3f3ca2c304f375f67d82fd75e3a41 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2eb50e37-d7c8-4e85-8439-9e8f590902f6_isrForCrosstalkSources_946477_56.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2eb50e37-d7c8-4e85-8439-9e8f590902f6_isrForCrosstalkSources_946477_56.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95aefdc2108c7525a34690ccf8fe4909 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/f50bf6a3-73fd-42ff-97e2-304abc5abbd3_isrForCrosstalkSources_946369_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/f50bf6a3-73fd-42ff-97e2-304abc5abbd3_isrForCrosstalkSources_946369_2.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 401e56550a5da6d755d67b40c9f0bc49 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/493150de-1be7-4067-b78d-4293badb890a_isrForCrosstalkSources_946369_49.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/493150de-1be7-4067-b78d-4293badb890a_isrForCrosstalkSources_946369_49.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b04753fb6bb21aa277aa7fdce18d2ea with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/aec6474f-09e1-47a0-92cd-7d104b8d8200_isrForCrosstalkSources_946371_22.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/aec6474f-09e1-47a0-92cd-7d104b8d8200_isrForCrosstalkSources_946371_22.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac06c02a791c96e5dc1d281ca5f0fc6c with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4b91b26b-0172-43d9-9dbe-b3de523d084f_isrForCrosstalkSources_946370_37.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4b91b26b-0172-43d9-9dbe-b3de523d084f_isrForCrosstalkSources_946370_37.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c6ee0f3c0d5d2c580491d2eda7c2ead with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/943a0be2-701c-4eec-94b6-c62dce6a04a0_isrForCrosstalkSources_946369_55.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/943a0be2-701c-4eec-94b6-c62dce6a04a0_isrForCrosstalkSources_946369_55.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 532221735ec79e5439276b36d5b7e672 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e0d6ba7d-276d-48a7-bd8b-1465e039e9a5_isrForCrosstalkSources_946371_12.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e0d6ba7d-276d-48a7-bd8b-1465e039e9a5_isrForCrosstalkSources_946371_12.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4186636829cef3b437d2dac4871f412 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1129e395-a0fb-458e-9ac3-4c0f2ae61792_isrForCrosstalkSources_946370_55.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1129e395-a0fb-458e-9ac3-4c0f2ae61792_isrForCrosstalkSources_946370_55.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 987dbaf40c1c61a98352715454b12f27 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e6eff6f4-3807-411b-8dcc-5a3fc4f7cafa_isrForCrosstalkSources_946477_52.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e6eff6f4-3807-411b-8dcc-5a3fc4f7cafa_isrForCrosstalkSources_946477_52.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 060aad550375a21e956a590b69a1a07d with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/8aec4174-a82a-44be-a73a-9d4cc2e3e5f3_isrForCrosstalkSources_946371_38.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/8aec4174-a82a-44be-a73a-9d4cc2e3e5f3_isrForCrosstalkSources_946371_38.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12913f364f13ac7915287c6ae84eb45d with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0e0d504b-d8e0-4e08-91ad-5d0ea671fcd9_isrForCrosstalkSources_946369_26.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0e0d504b-d8e0-4e08-91ad-5d0ea671fcd9_isrForCrosstalkSources_946369_26.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 972e38af65fba38af28d3bfe42c6fb62 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e160c802-fc45-4402-aa27-647f311b8cd9_isrForCrosstalkSources_946479_32.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e160c802-fc45-4402-aa27-647f311b8cd9_isrForCrosstalkSources_946479_32.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 381d33eb037882e4d655eca51f3f5849 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3eb306aa-7e6c-440f-b61f-c1c50b816cac_isrForCrosstalkSources_946479_13.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/3eb306aa-7e6c-440f-b61f-c1c50b816cac_isrForCrosstalkSources_946479_13.stderr 2024-03-13 23:14:14 proc_lsst.multi:146 [INFO] found job 20175 in provider local proc_lsst.multi INFO: found job 20175 in provider local parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16c453e73d9bc0d5468802bccde4b481 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/203c7f41-dfa6-4792-b97c-ddc47a626036_isrForCrosstalkSources_946371_39.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/203c7f41-dfa6-4792-b97c-ddc47a626036_isrForCrosstalkSources_946371_39.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18ff8746c3b2ba3d2afc65ec60975ccf with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2342a0e5-0f51-4dba-b889-685f38f41ed7_isrForCrosstalkSources_946369_23.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/2342a0e5-0f51-4dba-b889-685f38f41ed7_isrForCrosstalkSources_946369_23.stderr 2024-03-13 23:14:14 proc_lsst.multi:146 [INFO] found job 17025207 in provider astro proc_lsst.multi INFO: found job 17025207 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e5fbcb9b33f13d025edb92812163946 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c4ea9b5e-08e2-4769-80b2-a90c3d70f9d6_isrForCrosstalkSources_946370_21.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c4ea9b5e-08e2-4769-80b2-a90c3d70f9d6_isrForCrosstalkSources_946370_21.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b449066ad04d4b1222967baceff94de with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/23874a99-7c0b-4579-aed8-5be01192485e_isrForCrosstalkSources_946371_54.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/23874a99-7c0b-4579-aed8-5be01192485e_isrForCrosstalkSources_946371_54.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd6cabf7a148fd0ed918e9402ba468b7 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/8075932a-9bf6-47b7-8e54-a25318225675_isrForCrosstalkSources_946478_22.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/8075932a-9bf6-47b7-8e54-a25318225675_isrForCrosstalkSources_946478_22.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d453f78eb29a2564417a18b62057b39 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1f729181-d75e-4aae-945f-c5e6820d6f40_isrForCrosstalkSources_946369_12.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1f729181-d75e-4aae-945f-c5e6820d6f40_isrForCrosstalkSources_946369_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:14 proc_lsst.multi:146 [INFO] found job 17025208 in provider astro proc_lsst.multi INFO: found job 17025208 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7596471f318f226ae3fa67d3d66348fc with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c16fbf04-c857-4b2b-9fa9-4c345813db30_isrForCrosstalkSources_946370_33.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c16fbf04-c857-4b2b-9fa9-4c345813db30_isrForCrosstalkSources_946370_33.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b956c537b5a8381efba8646715e48db9 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/adfe5258-e5fd-4941-bd66-a10af1ff72fe_isrForCrosstalkSources_946479_18.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/adfe5258-e5fd-4941-bd66-a10af1ff72fe_isrForCrosstalkSources_946479_18.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e663b0813a6e40ecd5cc4dae1759e6f with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/46e53022-5a95-4b7b-a3e5-fd21e893c8e9_isrForCrosstalkSources_946371_9.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/46e53022-5a95-4b7b-a3e5-fd21e893c8e9_isrForCrosstalkSources_946371_9.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c11e07b059461caa95763b14331fea67 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/d54c9870-7ad0-4173-ba6a-cd8e820728f1_isrForCrosstalkSources_946479_15.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/d54c9870-7ad0-4173-ba6a-cd8e820728f1_isrForCrosstalkSources_946479_15.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 966f1330d44fcdd50d5433ac7e0bbacb with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ec55e126-1ca9-4c86-9d19-b9b6bdc7e03b_isrForCrosstalkSources_946478_47.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ec55e126-1ca9-4c86-9d19-b9b6bdc7e03b_isrForCrosstalkSources_946478_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:14 proc_lsst.multi:146 [INFO] found job 17025219 in provider astro proc_lsst.multi INFO: found job 17025219 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 956090925a7b290f53c614955fcf2d67 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5080a587-5cfd-481d-b69d-d937a879dc40_isrForCrosstalkSources_946369_15.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5080a587-5cfd-481d-b69d-d937a879dc40_isrForCrosstalkSources_946369_15.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3538d971fc8fa923eb84aa6e99298708 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/2fcf7b8d-9592-4023-b2ab-42e8c01ad3a1_isrForCrosstalkSources_946478_57.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/2fcf7b8d-9592-4023-b2ab-42e8c01ad3a1_isrForCrosstalkSources_946478_57.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a62363d4b06588cda926379bc2594eb3 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a721cfeb-78a0-480b-a044-1f2f9b467427_isrForCrosstalkSources_946371_2.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a721cfeb-78a0-480b-a044-1f2f9b467427_isrForCrosstalkSources_946371_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:14 proc_lsst.multi:146 [INFO] found job 17025220 in provider astro proc_lsst.multi INFO: found job 17025220 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d48d978ebc2abe6cc49eed7a980267e with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/24901007-50be-49aa-85b7-77e80b5437e9_isrForCrosstalkSources_946478_38.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/24901007-50be-49aa-85b7-77e80b5437e9_isrForCrosstalkSources_946478_38.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 178d02f8c71133075de0cdcf7763b689 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/ad8de4ba-ee5a-4a25-8fc1-54cbf84802dd_isrForCrosstalkSources_946370_28.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/ad8de4ba-ee5a-4a25-8fc1-54cbf84802dd_isrForCrosstalkSources_946370_28.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfd1baf1f84ccaa5c16f289059a057d6 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ff136d41-db11-4f25-95d6-c2269c7543ed_isrForCrosstalkSources_946478_53.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ff136d41-db11-4f25-95d6-c2269c7543ed_isrForCrosstalkSources_946478_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:14 proc_lsst.multi:146 [INFO] found job 17025221 in provider astro proc_lsst.multi INFO: found job 17025221 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f9128c90b094bbd5bb051d8323c60e8 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d439fdcc-7956-4b02-896f-46beb0224cfa_isrForCrosstalkSources_946478_52.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d439fdcc-7956-4b02-896f-46beb0224cfa_isrForCrosstalkSources_946478_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:15 proc_lsst.multi:146 [INFO] found job 17025222 in provider astro proc_lsst.multi INFO: found job 17025222 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e5dfa97281487dc7a07ef2c8208771 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ece1ba95-2494-4fc3-946e-8db001af3163_isrForCrosstalkSources_946371_4.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ece1ba95-2494-4fc3-946e-8db001af3163_isrForCrosstalkSources_946371_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:15 proc_lsst.multi:146 [INFO] found job 17025223 in provider astro proc_lsst.multi INFO: found job 17025223 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:15 proc_lsst.multi:146 [INFO] found job 17025224 in provider astro proc_lsst.multi INFO: found job 17025224 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:15 proc_lsst.multi:146 [INFO] found job 17025225 in provider astro proc_lsst.multi INFO: found job 17025225 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:15 proc_lsst.multi:146 [INFO] found job 17025226 in provider astro proc_lsst.multi INFO: found job 17025226 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7957f64a9123ce4da2bf4da88cf89cb with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0f9d31dd-25df-4b11-885a-d1631751be6c_isrForCrosstalkSources_946371_5.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/0f9d31dd-25df-4b11-885a-d1631751be6c_isrForCrosstalkSources_946371_5.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55c28a05be9fc29cf14c4b2ae7bfb7f9 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4be83675-b42d-441e-829c-1c922d325e12_isrForCrosstalkSources_946371_26.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4be83675-b42d-441e-829c-1c922d325e12_isrForCrosstalkSources_946371_26.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 231d9744286e64cc55805bed825d3db5 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/92ac0362-bfc6-45da-b076-1e448dd8b02b_isrForCrosstalkSources_946478_29.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/92ac0362-bfc6-45da-b076-1e448dd8b02b_isrForCrosstalkSources_946478_29.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76e35a737decc5a9d7f7eb9e55d7da81 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/39686566-2492-44c0-9956-85faaf14190a_isrForCrosstalkSources_946478_48.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/39686566-2492-44c0-9956-85faaf14190a_isrForCrosstalkSources_946478_48.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23b6eae0d64f43551132ae9b4b9572f3 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/a0f8da35-2411-4a6c-9b29-6319cc42c93d_isrForCrosstalkSources_946477_43.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/a0f8da35-2411-4a6c-9b29-6319cc42c93d_isrForCrosstalkSources_946477_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:15 proc_lsst.multi:146 [INFO] found job 17025227 in provider astro proc_lsst.multi INFO: found job 17025227 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93aafdf3f3af1c135100b786e0b6a45e with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/318c7003-9a8d-4470-b4c6-eeb72aec031a_isrForCrosstalkSources_946370_1.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/318c7003-9a8d-4470-b4c6-eeb72aec031a_isrForCrosstalkSources_946370_1.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f5d650eb1f2ac1e03ae574a84810e22 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1916d08c-9de0-462b-8767-2903a8977cab_isrForCrosstalkSources_946370_30.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/1916d08c-9de0-462b-8767-2903a8977cab_isrForCrosstalkSources_946370_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:15 proc_lsst.multi:146 [INFO] found job 17025228 in provider astro proc_lsst.multi INFO: found job 17025228 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e29283e842352299d037de494ab186d with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/054623cd-b992-4e65-8810-06b8a0bfeac5_isrForCrosstalkSources_946477_22.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/054623cd-b992-4e65-8810-06b8a0bfeac5_isrForCrosstalkSources_946477_22.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50bb37740ec0cba810193e81c9448460 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/df1f2592-4f90-4e24-b86f-831a7d7155c6_isrForCrosstalkSources_946479_10.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/df1f2592-4f90-4e24-b86f-831a7d7155c6_isrForCrosstalkSources_946479_10.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da565a8900725dab5b9d40f244e23595 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/38f3a9fe-d60b-4e8e-94ac-5d7d3a0fd784_isrForCrosstalkSources_946369_57.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/38f3a9fe-d60b-4e8e-94ac-5d7d3a0fd784_isrForCrosstalkSources_946369_57.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c481241dace558e205477641ec2ccc67 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f6462c8e-8706-45c5-b8ab-26f0fa28ef92_isrForCrosstalkSources_946477_1.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f6462c8e-8706-45c5-b8ab-26f0fa28ef92_isrForCrosstalkSources_946477_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:16 proc_lsst.multi:146 [INFO] found job 17025238 in provider astro proc_lsst.multi INFO: found job 17025238 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea2eca1e4547ba0acd9075a10e4af5bb with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/b3cbb415-9c40-4825-9488-524bb6b3adc6_isrForCrosstalkSources_946479_41.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/b3cbb415-9c40-4825-9488-524bb6b3adc6_isrForCrosstalkSources_946479_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:16 proc_lsst.multi:146 [INFO] found job 17025239 in provider astro proc_lsst.multi INFO: found job 17025239 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa88e4995a295c68332ffced2fc4e91d with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c55e9fe4-f220-4841-8c91-bab9c4131285_isrForCrosstalkSources_946370_49.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c55e9fe4-f220-4841-8c91-bab9c4131285_isrForCrosstalkSources_946370_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:16 proc_lsst.multi:146 [INFO] found job 17025240 in provider astro proc_lsst.multi INFO: found job 17025240 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:16 proc_lsst.multi:146 [INFO] found job 17025241 in provider astro proc_lsst.multi INFO: found job 17025241 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:16 proc_lsst.multi:146 [INFO] found job 17025242 in provider astro proc_lsst.multi INFO: found job 17025242 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4fe3378e8bbb8b073cb548057e2e20d with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/2d503654-2dcc-4109-a697-1635321f17a4_isrForCrosstalkSources_946370_42.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/2d503654-2dcc-4109-a697-1635321f17a4_isrForCrosstalkSources_946370_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:16 proc_lsst.multi:146 [INFO] found job 17025243 in provider astro proc_lsst.multi INFO: found job 17025243 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:17 proc_lsst.multi:146 [INFO] found job 17025244 in provider astro proc_lsst.multi INFO: found job 17025244 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c28b813a75b3e5a93eeb39a62ccc717b with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4411c94b-8a86-42aa-bce3-e65af3b65faa_isrForCrosstalkSources_946370_62.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/4411c94b-8a86-42aa-bce3-e65af3b65faa_isrForCrosstalkSources_946370_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:17 proc_lsst.multi:146 [INFO] found job 17025245 in provider astro proc_lsst.multi INFO: found job 17025245 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:17 proc_lsst.multi:146 [INFO] found job 17025246 in provider astro proc_lsst.multi INFO: found job 17025246 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:17 proc_lsst.multi:146 [INFO] found job 17025247 in provider astro proc_lsst.multi INFO: found job 17025247 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:17 proc_lsst.multi:146 [INFO] found job 17025248 in provider astro proc_lsst.multi INFO: found job 17025248 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:17 proc_lsst.multi:146 [INFO] found job 17025249 in provider astro proc_lsst.multi INFO: found job 17025249 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:17 proc_lsst.multi:146 [INFO] found job 17025250 in provider astro proc_lsst.multi INFO: found job 17025250 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b0fd091aff81de2659570aff7c77818 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/943e975d-23df-4d78-b1e6-85d507f3b92f_isrForCrosstalkSources_946479_20.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/943e975d-23df-4d78-b1e6-85d507f3b92f_isrForCrosstalkSources_946479_20.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2de7724d8d3840b9373372517b45a613 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/b1443ead-267b-44f4-9d24-53c685865d11_isrForCrosstalkSources_946371_58.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/b1443ead-267b-44f4-9d24-53c685865d11_isrForCrosstalkSources_946371_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:18 proc_lsst.multi:146 [INFO] found job 17025251 in provider astro proc_lsst.multi INFO: found job 17025251 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9350386febc29673f3c39599cca00b1a with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f66de3ff-c479-471d-ad26-f9eba3c363eb_isrForCrosstalkSources_946477_11.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/f66de3ff-c479-471d-ad26-f9eba3c363eb_isrForCrosstalkSources_946477_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:18 proc_lsst.multi:146 [INFO] found job 17025252 in provider astro proc_lsst.multi INFO: found job 17025252 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62b68de191b4b5c923e1dac6186bf681 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/9377d46a-47c4-41d4-95c1-b36039f0ae34_isrForCrosstalkSources_946371_30.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/9377d46a-47c4-41d4-95c1-b36039f0ae34_isrForCrosstalkSources_946371_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:18 proc_lsst.multi:146 [INFO] found job 17025253 in provider astro proc_lsst.multi INFO: found job 17025253 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b39e2749265c59e70c0bd49aa4f2f7fa with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ed9cabcc-b410-4acc-b101-cccb358ce7ea_isrForCrosstalkSources_946478_32.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ed9cabcc-b410-4acc-b101-cccb358ce7ea_isrForCrosstalkSources_946478_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:18 proc_lsst.multi:146 [INFO] found job 17025254 in provider astro proc_lsst.multi INFO: found job 17025254 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef64131f6ca7239f31c23b935aeeeabe with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0f848643-14a4-4c94-9141-ca99a5ac19df_isrForCrosstalkSources_946477_39.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0f848643-14a4-4c94-9141-ca99a5ac19df_isrForCrosstalkSources_946477_39.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a3b926b25fa396f25a466816f7cc4f5 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/070ffbe7-202a-43ff-beea-278992156675_isrForCrosstalkSources_946369_33.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/070ffbe7-202a-43ff-beea-278992156675_isrForCrosstalkSources_946369_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:18 proc_lsst.multi:146 [INFO] found job 17025255 in provider astro proc_lsst.multi INFO: found job 17025255 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025207,17025208,17025219,17025220,17025221,17025222,17025223,17025224,17025225,17025226,17025227,17025228,17025238,17025239,17025240,17025241,17025242,17025243,17025244,17025245,17025246,17025247,17025248,17025249,17025250,17025251,17025252,17025253,17025254,17025255' parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 087743680d7cf9a8d092b05486389ffd with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/d8d082e3-a038-42c6-b2e0-9a7fc5f8a688_isrForCrosstalkSources_946370_41.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/d8d082e3-a038-42c6-b2e0-9a7fc5f8a688_isrForCrosstalkSources_946370_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025255 R 17025248 R 17025249 R 17025250 R 17025251 R 17025252 R 17025253 R 17025254 R 17025238 R 17025239 R 17025240 R 17025241 R 17025242 R 17025243 R 17025244 R 17025245 R 17025246 R 17025247 R 17025219 R 17025220 R 17025221 R 17025222 R 17025223 R 17025224 R 17025225 R 17025226 R 17025227 R 17025228 R 17025207 R 17025208 R parsl.providers.slurm.slurm DEBUG: Updating job 17025255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025208 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:18 proc_lsst.multi:146 [INFO] found job 17025256 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025256 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86b022babf387003d829f9b5ab5ed2d8 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/02d5c662-0357-478c-b926-cc55bcae00b2_isrForCrosstalkSources_946370_59.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/02d5c662-0357-478c-b926-cc55bcae00b2_isrForCrosstalkSources_946370_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:19 proc_lsst.multi:146 [INFO] found job 17025257 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025257 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be7f603e1c5d44ccc6ca82bed4f42a9f with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/14e84241-1378-403c-82de-984954e19477_isrForCrosstalkSources_946479_17.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/14e84241-1378-403c-82de-984954e19477_isrForCrosstalkSources_946479_17.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3748b8acaab5c93ad23b77832291091 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/8ee8f177-9278-4a80-97ec-53ddbdf70229_isrForCrosstalkSources_946479_1.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/8ee8f177-9278-4a80-97ec-53ddbdf70229_isrForCrosstalkSources_946479_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:19 proc_lsst.multi:146 [INFO] found job 17025258 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025258 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b4ee988a11350d9973a7e84bfdd1dd8 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9390606e-ab8f-485e-b92b-8201aab65b0f_isrForCrosstalkSources_946479_38.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9390606e-ab8f-485e-b92b-8201aab65b0f_isrForCrosstalkSources_946479_38.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b375a85c53c6a22d4ec120529f338e0 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/72e78781-a358-486f-a823-4e2694e94abc_isrForCrosstalkSources_946371_46.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/72e78781-a358-486f-a823-4e2694e94abc_isrForCrosstalkSources_946371_46.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 653f6efead7e224dab59d4a12a6f9248 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c76e81aa-6eea-45e4-8925-0f1758202f20_isrForCrosstalkSources_946478_9.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c76e81aa-6eea-45e4-8925-0f1758202f20_isrForCrosstalkSources_946478_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:19 proc_lsst.multi:146 [INFO] found job 17025259 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025259 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6768d0ba3e97b3b2fef7bbd05e2588af with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/85998cdf-8d1f-4963-bfbb-a1ae1ef52b0a_isrForCrosstalkSources_946479_52.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/85998cdf-8d1f-4963-bfbb-a1ae1ef52b0a_isrForCrosstalkSources_946479_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:19 proc_lsst.multi:146 [INFO] found job 17025260 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025260 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025256,17025257,17025258,17025259,17025260' parsl.providers.slurm.slurm DEBUG: squeue returned 17025259 R 17025260 R 17025256 R 17025257 R 17025258 R parsl.providers.slurm.slurm DEBUG: Updating job 17025259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025256 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025258 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:19 proc_lsst.multi:146 [INFO] found job 17025261 in provider escience_a40 proc_lsst.multi INFO: found job 17025261 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d800f9c6dfbe0b5b1850d4c5e3d6197b with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/b9ec6b07-6758-404d-919c-76ef92cbabea_isrForCrosstalkSources_946370_38.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/b9ec6b07-6758-404d-919c-76ef92cbabea_isrForCrosstalkSources_946370_38.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 640935de9c96c74dca4dcc9d5d9f90cc with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/82d10d84-e7b5-401b-b1b4-2517a6f53543_isrForCrosstalkSources_946477_54.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/82d10d84-e7b5-401b-b1b4-2517a6f53543_isrForCrosstalkSources_946477_54.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d734e3a0a7d96c3ebd7eed0edad01ce4 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/cedfc79a-6854-4ad6-a119-c23ef7e2a762_isrForCrosstalkSources_946477_50.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/cedfc79a-6854-4ad6-a119-c23ef7e2a762_isrForCrosstalkSources_946477_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:19 proc_lsst.multi:146 [INFO] found job 17025262 in provider escience_a40 proc_lsst.multi INFO: found job 17025262 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:19 proc_lsst.multi:146 [INFO] found job 17025263 in provider escience_a40 proc_lsst.multi INFO: found job 17025263 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:20 proc_lsst.multi:146 [INFO] found job 17025264 in provider escience_a40 proc_lsst.multi INFO: found job 17025264 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dbe514078adbb785d1a3a16e754020f with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ee96815f-ce6d-4d09-9732-2cabc7897c8c_isrForCrosstalkSources_946478_19.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/ee96815f-ce6d-4d09-9732-2cabc7897c8c_isrForCrosstalkSources_946478_19.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e7cb5e60eea27b566d42b0f60bae3a5 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/dd804b3f-4545-4a62-818f-800cee989b4b_isrForCrosstalkSources_946479_60.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/dd804b3f-4545-4a62-818f-800cee989b4b_isrForCrosstalkSources_946479_60.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87fd3b0b180d36232adff9acf9bf1b53 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/d23c061d-4f40-4268-895a-7b99f44cfaf9_isrForCrosstalkSources_946479_53.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/d23c061d-4f40-4268-895a-7b99f44cfaf9_isrForCrosstalkSources_946479_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:20 proc_lsst.multi:146 [INFO] found job 17025265 in provider escience_a40 proc_lsst.multi INFO: found job 17025265 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37710ae27f9a862409bc1a6fe68fb3a0 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e5f29498-a8b5-413e-b5b1-56a2540b5c59_isrForCrosstalkSources_946371_52.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/e5f29498-a8b5-413e-b5b1-56a2540b5c59_isrForCrosstalkSources_946371_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:20 proc_lsst.multi:146 [INFO] found job 17025266 in provider escience_a40 proc_lsst.multi INFO: found job 17025266 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e613732618c712ae1b96d2d3d042c00 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1973ce09-ba36-44d5-9ecc-3f76e64d6fb1_isrForCrosstalkSources_946369_54.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/1973ce09-ba36-44d5-9ecc-3f76e64d6fb1_isrForCrosstalkSources_946369_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:20 proc_lsst.multi:146 [INFO] found job 17025267 in provider escience_a40 proc_lsst.multi INFO: found job 17025267 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:20 proc_lsst.multi:146 [INFO] found job 17025268 in provider escience_a40 proc_lsst.multi INFO: found job 17025268 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:20 proc_lsst.multi:146 [INFO] found job 17025269 in provider escience_a40 proc_lsst.multi INFO: found job 17025269 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:20 proc_lsst.multi:146 [INFO] found job 17025270 in provider escience_a40 proc_lsst.multi INFO: found job 17025270 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:21 proc_lsst.multi:146 [INFO] found job 17025271 in provider escience_a40 proc_lsst.multi INFO: found job 17025271 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:21 proc_lsst.multi:146 [INFO] found job 17025272 in provider escience_a40 proc_lsst.multi INFO: found job 17025272 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:21 proc_lsst.multi:146 [INFO] found job 17025273 in provider escience_a40 proc_lsst.multi INFO: found job 17025273 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:21 proc_lsst.multi:146 [INFO] found job 17025274 in provider escience_a40 proc_lsst.multi INFO: found job 17025274 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:21 proc_lsst.multi:146 [INFO] found job 17025275 in provider escience_a40 proc_lsst.multi INFO: found job 17025275 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:21 proc_lsst.multi:146 [INFO] found job 17025276 in provider escience_a40 proc_lsst.multi INFO: found job 17025276 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:22 proc_lsst.multi:146 [INFO] found job 17025277 in provider escience_a40 proc_lsst.multi INFO: found job 17025277 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:22 proc_lsst.multi:146 [INFO] found job 17025278 in provider escience_a40 proc_lsst.multi INFO: found job 17025278 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:22 proc_lsst.multi:146 [INFO] found job 17025279 in provider escience_a40 proc_lsst.multi INFO: found job 17025279 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:22 proc_lsst.multi:146 [INFO] found job 17025280 in provider escience_a40 proc_lsst.multi INFO: found job 17025280 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025261,17025262,17025263,17025264,17025265,17025266,17025267,17025268,17025269,17025270,17025271,17025272,17025273,17025274,17025275,17025276,17025277,17025278,17025279,17025280' parsl.providers.slurm.slurm DEBUG: squeue returned 17025278 R 17025279 R 17025280 R 17025270 R 17025271 R 17025272 R 17025273 R 17025274 R 17025275 R 17025276 R 17025277 R 17025261 R 17025262 R 17025263 R 17025264 R 17025265 R 17025266 R 17025267 R 17025268 R 17025269 R parsl.providers.slurm.slurm DEBUG: Updating job 17025278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025269 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:22 proc_lsst.multi:146 [INFO] found job 17025281 in provider checkpoint proc_lsst.multi INFO: found job 17025281 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:22 proc_lsst.multi:146 [INFO] found job 17025282 in provider checkpoint proc_lsst.multi INFO: found job 17025282 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:22 proc_lsst.multi:146 [INFO] found job 17025283 in provider checkpoint proc_lsst.multi INFO: found job 17025283 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:23 proc_lsst.multi:146 [INFO] found job 17025284 in provider checkpoint proc_lsst.multi INFO: found job 17025284 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:23 proc_lsst.multi:146 [INFO] found job 17025285 in provider checkpoint proc_lsst.multi INFO: found job 17025285 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:23 proc_lsst.multi:146 [INFO] found job 17025286 in provider checkpoint proc_lsst.multi INFO: found job 17025286 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:23 proc_lsst.multi:146 [INFO] found job 17025287 in provider checkpoint proc_lsst.multi INFO: found job 17025287 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:23 proc_lsst.multi:146 [INFO] found job 17025288 in provider checkpoint proc_lsst.multi INFO: found job 17025288 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:23 proc_lsst.multi:146 [INFO] found job 17025289 in provider checkpoint proc_lsst.multi INFO: found job 17025289 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:24 proc_lsst.multi:146 [INFO] found job 17025290 in provider checkpoint proc_lsst.multi INFO: found job 17025290 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:24 proc_lsst.multi:146 [INFO] found job 17025291 in provider checkpoint proc_lsst.multi INFO: found job 17025291 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:24 proc_lsst.multi:146 [INFO] found job 17025292 in provider checkpoint proc_lsst.multi INFO: found job 17025292 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:24 proc_lsst.multi:146 [INFO] found job 17025293 in provider checkpoint proc_lsst.multi INFO: found job 17025293 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:24 proc_lsst.multi:146 [INFO] found job 17025294 in provider checkpoint proc_lsst.multi INFO: found job 17025294 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:24 proc_lsst.multi:146 [INFO] found job 17025295 in provider checkpoint proc_lsst.multi INFO: found job 17025295 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:25 proc_lsst.multi:146 [INFO] found job 17025296 in provider checkpoint proc_lsst.multi INFO: found job 17025296 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:25 proc_lsst.multi:146 [INFO] found job 17025297 in provider checkpoint proc_lsst.multi INFO: found job 17025297 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:25 proc_lsst.multi:146 [INFO] found job 17025298 in provider checkpoint proc_lsst.multi INFO: found job 17025298 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:25 proc_lsst.multi:146 [INFO] found job 17025299 in provider checkpoint proc_lsst.multi INFO: found job 17025299 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:25 proc_lsst.multi:146 [INFO] found job 17025300 in provider checkpoint proc_lsst.multi INFO: found job 17025300 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:25 proc_lsst.multi:146 [INFO] found job 17025301 in provider checkpoint proc_lsst.multi INFO: found job 17025301 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:25 proc_lsst.multi:146 [INFO] found job 17025302 in provider checkpoint proc_lsst.multi INFO: found job 17025302 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:26 proc_lsst.multi:146 [INFO] found job 17025303 in provider checkpoint proc_lsst.multi INFO: found job 17025303 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:26 proc_lsst.multi:146 [INFO] found job 17025304 in provider checkpoint proc_lsst.multi INFO: found job 17025304 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:26 proc_lsst.multi:146 [INFO] found job 17025305 in provider checkpoint proc_lsst.multi INFO: found job 17025305 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:26 proc_lsst.multi:146 [INFO] found job 17025306 in provider checkpoint proc_lsst.multi INFO: found job 17025306 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:27 proc_lsst.multi:146 [INFO] found job 17025307 in provider checkpoint proc_lsst.multi INFO: found job 17025307 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8abe8e6f72521e7b965b9857ca579b32 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/bb8cb57a-900f-436c-bc4a-ebad93fbc1d9_isrForCrosstalkSources_946479_47.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/bb8cb57a-900f-436c-bc4a-ebad93fbc1d9_isrForCrosstalkSources_946479_47.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72ec0e4cb61b9538026b6e545e34c398 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/e4c6ba2d-2461-40f2-8835-8e01c5c2d5fe_isrForCrosstalkSources_946369_60.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/e4c6ba2d-2461-40f2-8835-8e01c5c2d5fe_isrForCrosstalkSources_946369_60.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8696d6ae1c7dad45da3dffb28af42d0 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a163ec42-2944-4154-ad66-f61a381d8af8_isrForCrosstalkSources_946478_11.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/a163ec42-2944-4154-ad66-f61a381d8af8_isrForCrosstalkSources_946478_11.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24569e3dd3b483910a4d171d3a25992f with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/de4239e4-e95a-4945-9100-d1bed240cb57_isrForCrosstalkSources_946477_55.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/de4239e4-e95a-4945-9100-d1bed240cb57_isrForCrosstalkSources_946477_55.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0f203a8b1355c728e2f965643392e5a with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e7acee43-3b35-4a1d-a9f8-59f8fbf43c8f_isrForCrosstalkSources_946477_57.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/e7acee43-3b35-4a1d-a9f8-59f8fbf43c8f_isrForCrosstalkSources_946477_57.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8023ed8bb0bfdfe5a3960f0d8d7b79e8 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/89850c1b-32b9-4b42-8edd-c667098c738e_isrForCrosstalkSources_946479_42.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/89850c1b-32b9-4b42-8edd-c667098c738e_isrForCrosstalkSources_946479_42.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a68202499f200ad7ac53afa7364397d4 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/245f0652-0c6e-418c-be87-3c4664d9393c_isrForCrosstalkSources_946477_46.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/245f0652-0c6e-418c-be87-3c4664d9393c_isrForCrosstalkSources_946477_46.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c884fb10fb9e1f9b05b3810be37c9e69 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c944ff9a-9be0-4a8f-aa66-567f7d8130b2_isrForCrosstalkSources_946370_12.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/c944ff9a-9be0-4a8f-aa66-567f7d8130b2_isrForCrosstalkSources_946370_12.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1be254db775430ae397ad7e792066056 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f552c9ad-f064-43cc-8965-8868990caecd_isrForCrosstalkSources_946478_34.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f552c9ad-f064-43cc-8965-8868990caecd_isrForCrosstalkSources_946478_34.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57611ab4035bc9cb5fb000b53805d249 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/da4a0698-6e00-457a-ae1f-63be6ba1d1d4_isrForCrosstalkSources_946477_10.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/da4a0698-6e00-457a-ae1f-63be6ba1d1d4_isrForCrosstalkSources_946477_10.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bd4ffc22541ee4d7b52dac78158c976 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/3a1226b4-73ba-476c-8d6b-dc5316e6f26c_isrForCrosstalkSources_946369_59.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/3a1226b4-73ba-476c-8d6b-dc5316e6f26c_isrForCrosstalkSources_946369_59.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b93b891f4c2fa9d1e497b16655d86a9 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/cb93b289-70be-4aa6-950b-3a54e8146089_isrForCrosstalkSources_946371_45.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/cb93b289-70be-4aa6-950b-3a54e8146089_isrForCrosstalkSources_946371_45.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acfe884836470e3e93e9738a592277b6 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/191142ba-d40c-40b4-95b9-7349aed7383e_isrForCrosstalkSources_946369_14.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/191142ba-d40c-40b4-95b9-7349aed7383e_isrForCrosstalkSources_946369_14.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46bdd7fff8e357b9728af7c0b474b30a with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0ce39571-32a2-45ae-9cdb-d72e42dd606a_isrForCrosstalkSources_946370_35.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0ce39571-32a2-45ae-9cdb-d72e42dd606a_isrForCrosstalkSources_946370_35.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d9622b43925afbb24d364674eead87e with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/bfe18b99-ccf6-4952-8484-520478310adc_isrForCrosstalkSources_946371_21.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/bfe18b99-ccf6-4952-8484-520478310adc_isrForCrosstalkSources_946371_21.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 902692b391601fc0ba6312c7f2f56699 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/884d5dcb-d026-451c-a80d-36cbb2e9c4bc_isrForCrosstalkSources_946478_2.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/884d5dcb-d026-451c-a80d-36cbb2e9c4bc_isrForCrosstalkSources_946478_2.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96781323d32c123f5f0d02228b528641 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3a7ba5dd-1f7d-46da-953a-abbb873646da_isrForCrosstalkSources_946477_53.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/3a7ba5dd-1f7d-46da-953a-abbb873646da_isrForCrosstalkSources_946477_53.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72fda2ad85984b019a29e23c71631570 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c7bfa312-37f0-4407-967c-cab8f4bc4cfe_isrForCrosstalkSources_946369_29.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c7bfa312-37f0-4407-967c-cab8f4bc4cfe_isrForCrosstalkSources_946369_29.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78ea635b0b9a74bdff5d31d7546bec91 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/65045e18-5aeb-4398-bb5f-96375df9a87f_isrForCrosstalkSources_946477_31.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/65045e18-5aeb-4398-bb5f-96375df9a87f_isrForCrosstalkSources_946477_31.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54599db3192529d3745c2923930f9e7d with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/5adc7c1e-1be2-491e-852a-7a11a7480c69_isrForCrosstalkSources_946370_18.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/5adc7c1e-1be2-491e-852a-7a11a7480c69_isrForCrosstalkSources_946370_18.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61495bacf2a0878200e490bd57b5f4ba with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/73e50b04-da2e-48b4-b42e-1db50f41c7ce_isrForCrosstalkSources_946477_8.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/73e50b04-da2e-48b4-b42e-1db50f41c7ce_isrForCrosstalkSources_946477_8.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58c55f383ca7087f69e01e204c545de5 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e3c58c92-78a5-4d76-90b3-ed512cd9199d_isrForCrosstalkSources_946370_11.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e3c58c92-78a5-4d76-90b3-ed512cd9199d_isrForCrosstalkSources_946370_11.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed4c21ae0504d5e86f9816eff31f868e with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c2563919-8fb8-437d-91fa-88cc918c61c4_isrForCrosstalkSources_946369_43.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c2563919-8fb8-437d-91fa-88cc918c61c4_isrForCrosstalkSources_946369_43.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5754f8f54bc6759f2a7041231150ed3 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/4941ba6d-554a-4321-a8b1-f3adcb4614bd_isrForCrosstalkSources_946479_50.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/4941ba6d-554a-4321-a8b1-f3adcb4614bd_isrForCrosstalkSources_946479_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:27 proc_lsst.multi:146 [INFO] found job 17025308 in provider checkpoint proc_lsst.multi INFO: found job 17025308 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:27 proc_lsst.multi:146 [INFO] found job 17025309 in provider checkpoint proc_lsst.multi INFO: found job 17025309 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:27 proc_lsst.multi:146 [INFO] found job 17025310 in provider checkpoint proc_lsst.multi INFO: found job 17025310 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:27 proc_lsst.multi:146 [INFO] found job 17025311 in provider checkpoint proc_lsst.multi INFO: found job 17025311 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:28 proc_lsst.multi:146 [INFO] found job 17025312 in provider checkpoint proc_lsst.multi INFO: found job 17025312 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:28 proc_lsst.multi:146 [INFO] found job 17025313 in provider checkpoint proc_lsst.multi INFO: found job 17025313 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:28 proc_lsst.multi:146 [INFO] found job 17025314 in provider checkpoint proc_lsst.multi INFO: found job 17025314 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:28 proc_lsst.multi:146 [INFO] found job 17025315 in provider checkpoint proc_lsst.multi INFO: found job 17025315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:28 proc_lsst.multi:146 [INFO] found job 17025316 in provider checkpoint proc_lsst.multi INFO: found job 17025316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97a51c5d759b59fe97274de9ee9f32f1 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/437c6410-6157-4ed6-8138-11fb1f5bbc4c_isrForCrosstalkSources_946478_27.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/437c6410-6157-4ed6-8138-11fb1f5bbc4c_isrForCrosstalkSources_946478_27.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c69892a95914708c03b476e8ba6466d with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6bf367fb-8022-4d9a-9408-2b41bc75de0f_isrForCrosstalkSources_946478_8.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/6bf367fb-8022-4d9a-9408-2b41bc75de0f_isrForCrosstalkSources_946478_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING 2024-03-13 23:14:28 proc_lsst.multi:146 [INFO] found job 17025317 in provider checkpoint proc_lsst.multi INFO: found job 17025317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17025281,17025282,17025283,17025284,17025285,17025286,17025287,17025288,17025289,17025290,17025291,17025292,17025293,17025294,17025295,17025296,17025297,17025298,17025299,17025300,17025301,17025302,17025303,17025304,17025305,17025306,17025307,17025308,17025309,17025310,17025311,17025312,17025313,17025314,17025315,17025316,17025317' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b3f933242552c9ac4e3c951eca496eb with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9cafbe2b-7970-4363-bdf0-fd940f880fd4_isrForCrosstalkSources_946477_19.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/9cafbe2b-7970-4363-bdf0-fd940f880fd4_isrForCrosstalkSources_946477_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17025309 PD 17025310 PD 17025311 PD 17025312 PD 17025313 PD 17025314 PD 17025315 PD 17025316 PD 17025317 PD 17025290 R 17025291 R 17025292 R 17025293 R 17025294 R 17025295 R 17025296 R 17025297 R 17025298 R 17025299 R 17025300 R 17025301 R 17025302 R 17025303 R 17025304 R 17025305 R 17025306 R 17025307 R 17025308 R 17025281 R 17025282 R 17025283 R 17025284 R 17025285 R 17025286 R 17025287 R 17025288 R 17025289 R parsl.providers.slurm.slurm DEBUG: Updating job 17025309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17025290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17025289 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 69 parsl.dataflow.strategy DEBUG: Executor multi has 69 active tasks, 84/9 running/pending blocks, and 281 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 69 parsl.dataflow.strategy DEBUG: Executor multi has 69 active tasks, 84/9 running/pending blocks, and 281 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98c5e79ba7da44f51719959ba0286dd2 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0b32cca8-aa31-4f6e-937f-6fb76e202799_isrForCrosstalkSources_946370_17.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0b32cca8-aa31-4f6e-937f-6fb76e202799_isrForCrosstalkSources_946370_17.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27da058f123b21213b8ee47f4214f2f5 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/5462ca9c-5499-4849-96dd-a0da2da2f0a1_isrForCrosstalkSources_946371_23.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/5462ca9c-5499-4849-96dd-a0da2da2f0a1_isrForCrosstalkSources_946371_23.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5b8cd0d0c12418b9147f6717f604875 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0de15106-b63d-450c-a4d0-d6272104c24c_isrForCrosstalkSources_946369_4.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0de15106-b63d-450c-a4d0-d6272104c24c_isrForCrosstalkSources_946369_4.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dddf8482ec032cd967049c21c069a82f with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/89ddeb0d-aabf-4e6b-81c3-18f2d95ef747_isrForCrosstalkSources_946478_33.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/89ddeb0d-aabf-4e6b-81c3-18f2d95ef747_isrForCrosstalkSources_946478_33.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 755f7b1cd27454050995c7f0b00d8475 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/500c259e-2506-4bad-b0b0-9bdfd403cd42_isrForCrosstalkSources_946371_33.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/500c259e-2506-4bad-b0b0-9bdfd403cd42_isrForCrosstalkSources_946371_33.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2adb9f2308fe0e729194f64ee262c417 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9d0fa824-1bd1-48c7-bb59-cb557a92fca4_isrForCrosstalkSources_946479_36.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9d0fa824-1bd1-48c7-bb59-cb557a92fca4_isrForCrosstalkSources_946479_36.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4a5eb5bad5ccd94261df586821cf17a with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2efbc69a-2f55-40a2-9f13-f44ad31badcb_isrForCrosstalkSources_946477_16.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/2efbc69a-2f55-40a2-9f13-f44ad31badcb_isrForCrosstalkSources_946477_16.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 086e764aee346c14512ec50d845c8d8a with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/5f3d473e-6e78-4900-baba-e592277452be_isrForCrosstalkSources_946479_54.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/5f3d473e-6e78-4900-baba-e592277452be_isrForCrosstalkSources_946479_54.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd5b4fcd972d7da8424075bce81e24f0 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e56103f2-84a1-4f72-ac6b-28886b5c171b_isrForCrosstalkSources_946478_46.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e56103f2-84a1-4f72-ac6b-28886b5c171b_isrForCrosstalkSources_946478_46.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae288a57a036fcb16b79455627cfad2e with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/df5a4d78-a8b7-4a2f-ab74-1a5575f4a135_isrForCrosstalkSources_946478_15.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/df5a4d78-a8b7-4a2f-ab74-1a5575f4a135_isrForCrosstalkSources_946478_15.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6257e4f083310a0ebcc1394c2c2193c2 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0b588a42-f22d-42e9-a2a7-4ca0a28163aa_isrForCrosstalkSources_946477_42.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/0b588a42-f22d-42e9-a2a7-4ca0a28163aa_isrForCrosstalkSources_946477_42.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 543e301ade364e67df0b46954edcbabe with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/16acc3fb-6757-4f9f-b3a5-b62aa6204dd8_isrForCrosstalkSources_946369_11.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/16acc3fb-6757-4f9f-b3a5-b62aa6204dd8_isrForCrosstalkSources_946369_11.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8594082aac58f265735688b37fcffab with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/916f6a15-2932-4ed0-aa6a-6a19034c4451_isrForCrosstalkSources_946371_25.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/916f6a15-2932-4ed0-aa6a-6a19034c4451_isrForCrosstalkSources_946371_25.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0606e30db141bc970f796fd0d6a55a28 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/886983e1-fbac-441e-a14e-b161b93e61f7_isrForCrosstalkSources_946479_48.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/886983e1-fbac-441e-a14e-b161b93e61f7_isrForCrosstalkSources_946479_48.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f3e1f85c75827253d30f55135919dc3 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/930f0dcb-04ac-424b-b7ef-11343d9b5068_isrForCrosstalkSources_946370_7.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/930f0dcb-04ac-424b-b7ef-11343d9b5068_isrForCrosstalkSources_946370_7.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57f2b55863a22c0e78d29682c9b8a20c with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0974e617-a519-4278-8b1b-5bbd05b90efc_isrForCrosstalkSources_946370_31.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/0974e617-a519-4278-8b1b-5bbd05b90efc_isrForCrosstalkSources_946370_31.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6ae5e84e4a7aca1c05d32a7b173ee62 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/799d634c-5afe-4c6d-94ab-d3fa65bedf48_isrForCrosstalkSources_946371_53.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/799d634c-5afe-4c6d-94ab-d3fa65bedf48_isrForCrosstalkSources_946371_53.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a49f43352b47c3e1e8ca5fe7d4577937 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/864d16b7-d0a9-48bf-89fa-8ae86edabc9b_isrForCrosstalkSources_946477_38.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/864d16b7-d0a9-48bf-89fa-8ae86edabc9b_isrForCrosstalkSources_946477_38.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e709484a362bbdaf0fe3ff438a6f20b with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4722ebf1-fea5-46e5-a865-102bec81638f_isrForCrosstalkSources_946371_51.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/4722ebf1-fea5-46e5-a865-102bec81638f_isrForCrosstalkSources_946371_51.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aaef821033b6a030a208cdc83b31154 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/effcce2b-94d1-4590-980e-0579221185fe_isrForCrosstalkSources_946371_29.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/effcce2b-94d1-4590-980e-0579221185fe_isrForCrosstalkSources_946371_29.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb1adca75f49450014ebd1693e059c80 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f5b877fe-653a-4359-a0e8-38d7462ef016_isrForCrosstalkSources_946478_62.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/f5b877fe-653a-4359-a0e8-38d7462ef016_isrForCrosstalkSources_946478_62.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03b09aac3fc1f878709ada863248d1c2 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0b8e9775-51f4-4d38-ab9d-671523f1457d_isrForCrosstalkSources_946369_51.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0b8e9775-51f4-4d38-ab9d-671523f1457d_isrForCrosstalkSources_946369_51.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51483f16d9762dd1b9a53f5028fbc867 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/d5de5f7c-a337-47f2-ae9c-e673863caf82_isrForCrosstalkSources_946477_4.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/d5de5f7c-a337-47f2-ae9c-e673863caf82_isrForCrosstalkSources_946477_4.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b69b3f6a6ab55deb0e428d91a4b7d959 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0aa0b2c3-3117-4e16-9dbd-ccdf8d62ce79_isrForCrosstalkSources_946369_18.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/0aa0b2c3-3117-4e16-9dbd-ccdf8d62ce79_isrForCrosstalkSources_946369_18.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94f2c88b389f6e8ab4326b9c776cbae5 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c6ac568d-4119-4774-8701-d27fedc73b5c_isrForCrosstalkSources_946478_20.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/c6ac568d-4119-4774-8701-d27fedc73b5c_isrForCrosstalkSources_946478_20.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc3fd0ff1caa5f019671869c8485ae58 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ab9bbd24-8d5c-4935-9127-8ec15b8ff56b_isrForCrosstalkSources_946371_37.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ab9bbd24-8d5c-4935-9127-8ec15b8ff56b_isrForCrosstalkSources_946371_37.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb5ddf7b21b650f241d0d8886c5553b7 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/de1c7f11-65e9-4722-aae1-4aec850c9df0_isrForCrosstalkSources_946370_13.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/de1c7f11-65e9-4722-aae1-4aec850c9df0_isrForCrosstalkSources_946370_13.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ba6963265e007f036f1cd9c922a8bd3 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/de06b344-d3fc-4f87-895c-dae03ae7f2de_isrForCrosstalkSources_946371_17.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/de06b344-d3fc-4f87-895c-dae03ae7f2de_isrForCrosstalkSources_946371_17.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20c49c427971c2471ee0f496f89b926a with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5ce7c244-0ec1-4e2b-bcda-75a07013fde3_isrForCrosstalkSources_946369_17.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/5ce7c244-0ec1-4e2b-bcda-75a07013fde3_isrForCrosstalkSources_946369_17.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1470c63c1a6a7c5e52150c836a8981cc with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/06238329-c8d4-41f4-8801-b41e3103d7ae_isrForCrosstalkSources_946477_51.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/06238329-c8d4-41f4-8801-b41e3103d7ae_isrForCrosstalkSources_946477_51.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1568864c0ee11d1d68617153fbebb77 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e420ae37-bc96-4521-8dd2-16cfaac8b648_isrForCrosstalkSources_946370_45.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/e420ae37-bc96-4521-8dd2-16cfaac8b648_isrForCrosstalkSources_946370_45.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae0408a48f6748a56aa110a18a7ce425 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9c0d45a0-83cd-4c1a-a57f-ae625a76267c_isrForCrosstalkSources_946479_28.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/9c0d45a0-83cd-4c1a-a57f-ae625a76267c_isrForCrosstalkSources_946479_28.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d084043d4e4093659a248ce7a8faf8ad with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/35f5059f-10e1-4cca-8652-8b3bbd68190e_isrForCrosstalkSources_946477_24.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/35f5059f-10e1-4cca-8652-8b3bbd68190e_isrForCrosstalkSources_946477_24.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9e0b6d4359b8102e81f19a321298028 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/572daf73-dd8d-47bd-88fc-db44375d113c_isrForCrosstalkSources_946369_47.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/572daf73-dd8d-47bd-88fc-db44375d113c_isrForCrosstalkSources_946369_47.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e24314ce4e4dcfa776fc77402f3bd6b with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6ba28200-cb11-488c-9292-2986a14c42c1_isrForCrosstalkSources_946371_42.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/6ba28200-cb11-488c-9292-2986a14c42c1_isrForCrosstalkSources_946371_42.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5473b02308c764a4498b9868903be4e with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/699773c6-7411-45db-8a10-cdb3b84b409a_isrForCrosstalkSources_946370_60.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/699773c6-7411-45db-8a10-cdb3b84b409a_isrForCrosstalkSources_946370_60.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a405fbdee86f98223cbe933ded817ebd with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ed80366f-51aa-447e-9451-ee70b08cbb54_isrForCrosstalkSources_946371_35.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ed80366f-51aa-447e-9451-ee70b08cbb54_isrForCrosstalkSources_946371_35.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71d469a7390f6538fa779efa58e9b4b6 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/5ec308cb-2547-4858-af13-09282b021cab_isrForCrosstalkSources_946478_45.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/5ec308cb-2547-4858-af13-09282b021cab_isrForCrosstalkSources_946478_45.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c01c66e7f24475efc5c2546c84b601ec with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/5c81372b-bcec-462e-8804-105d5a26c5ef_isrForCrosstalkSources_946477_37.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/5c81372b-bcec-462e-8804-105d5a26c5ef_isrForCrosstalkSources_946477_37.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e04c5d658d7d187cc6ea4bdb31228aa with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/071180af-de8c-48fa-86dc-1cf06a4c87b1_isrForCrosstalkSources_946477_12.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/071180af-de8c-48fa-86dc-1cf06a4c87b1_isrForCrosstalkSources_946477_12.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c0cf0e295b0c45a513eed8e3139226e with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/78c8ef83-3960-4cec-9f3c-c3b534c4afd8_isrForCrosstalkSources_946477_36.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/78c8ef83-3960-4cec-9f3c-c3b534c4afd8_isrForCrosstalkSources_946477_36.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4768023d39f696b929d234bda102459 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/33589ec4-7abd-4b30-841f-857ed4fb25d8_isrForCrosstalkSources_946370_10.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/33589ec4-7abd-4b30-841f-857ed4fb25d8_isrForCrosstalkSources_946370_10.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 231fd3afd090cd8948bb49982bcd5842 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/17c19e79-1a23-4b9f-95c4-c39ba0c75a5f_isrForCrosstalkSources_946477_45.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/17c19e79-1a23-4b9f-95c4-c39ba0c75a5f_isrForCrosstalkSources_946477_45.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8382df44d02fd041648d54e2d064f8f with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/373279ca-4da7-4549-8c87-9b9c6cac8514_isrForCrosstalkSources_946370_58.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/373279ca-4da7-4549-8c87-9b9c6cac8514_isrForCrosstalkSources_946370_58.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bd0c9ebb95a7cae265a153ae12594e7 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/dcfb5ad7-9d8a-457c-9b92-ffc652c63342_isrForCrosstalkSources_946371_34.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/dcfb5ad7-9d8a-457c-9b92-ffc652c63342_isrForCrosstalkSources_946371_34.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 24 parsl.dataflow.strategy DEBUG: Executor multi has 24 active tasks, 84/9 running/pending blocks, and 281 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0ae244df63964e06b1630b1996c3782 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/620bb6be-b9f6-4925-b1f9-e964d8074e76_isrForCrosstalkSources_946371_57.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/620bb6be-b9f6-4925-b1f9-e964d8074e76_isrForCrosstalkSources_946371_57.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e390e7a9fd32b6640a51ec20a9983e62 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/6b41bb53-0658-4bff-8a9d-b9139fe7d7c1_isrForCrosstalkSources_946370_53.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/6b41bb53-0658-4bff-8a9d-b9139fe7d7c1_isrForCrosstalkSources_946370_53.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3efc0ade6783d5291e9a566322bd83d7 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/fc31c5de-9e36-4071-91a7-f84fd169ebf2_isrForCrosstalkSources_946369_10.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/fc31c5de-9e36-4071-91a7-f84fd169ebf2_isrForCrosstalkSources_946369_10.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59fc8b3e49675beba8a852d7a416e255 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/bc48ae07-b84e-4227-82d7-c53927a8f68d_isrForCrosstalkSources_946479_29.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/bc48ae07-b84e-4227-82d7-c53927a8f68d_isrForCrosstalkSources_946479_29.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc061ea5dcd34e8fb8f68755d52e31c9 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e208afd9-fd45-42f2-b4e8-115d715ee801_isrForCrosstalkSources_946478_25.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e208afd9-fd45-42f2-b4e8-115d715ee801_isrForCrosstalkSources_946478_25.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e4d6097aa3116ca6b4eec7fed623b55 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d09ab204-d420-430b-ae7b-e0d353281f5a_isrForCrosstalkSources_946478_17.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/d09ab204-d420-430b-ae7b-e0d353281f5a_isrForCrosstalkSources_946478_17.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96f168409d97afc6ee7f770caede7a97 with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e0a4edcc-dcba-4266-a788-f6a00251347f_isrForCrosstalkSources_946478_24.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e0a4edcc-dcba-4266-a788-f6a00251347f_isrForCrosstalkSources_946478_24.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a291c72a44c47b5e924a531749778d2 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a1f89a77-8564-4d95-bdce-f1ce9b0e4321_isrForCrosstalkSources_946371_13.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/a1f89a77-8564-4d95-bdce-f1ce9b0e4321_isrForCrosstalkSources_946371_13.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b0a8d1bcdc61ac3c69ffc87097716f1 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/1af49b2d-41c8-48c2-8e61-b7923a5d48ab_isrForCrosstalkSources_946478_14.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/1af49b2d-41c8-48c2-8e61-b7923a5d48ab_isrForCrosstalkSources_946478_14.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 476a0ce445c765ca793e6279e822c651 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e33ed253-aa02-44fb-a12b-f67992d60eab_isrForCrosstalkSources_946479_46.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/e33ed253-aa02-44fb-a12b-f67992d60eab_isrForCrosstalkSources_946479_46.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b50265b860303b47407350f6b81961a3 with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/48acea09-0831-4d98-853d-050a3d6989e5_isrForCrosstalkSources_946478_39.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/48acea09-0831-4d98-853d-050a3d6989e5_isrForCrosstalkSources_946478_39.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77e3d37a102d513df0140c6b95ee8d42 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/7fcb135a-af14-4962-99b9-0767e28403ac_isrForCrosstalkSources_946478_30.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/7fcb135a-af14-4962-99b9-0767e28403ac_isrForCrosstalkSources_946478_30.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2126518d67dcba773b9599e0ea45e641 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/581c508b-6bb2-4a2d-aa43-7d9963a31541_isrForCrosstalkSources_946370_43.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/581c508b-6bb2-4a2d-aa43-7d9963a31541_isrForCrosstalkSources_946370_43.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e25af13471a7bab3c94d3d5c1e6e32d1 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6d0a3fa4-5c26-49a3-978f-d2ec17c15984_isrForCrosstalkSources_946477_7.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946477/6d0a3fa4-5c26-49a3-978f-d2ec17c15984_isrForCrosstalkSources_946477_7.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a0451d79540f897d02636937151b2d9 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e0075896-d184-477b-bcba-176d6edeeac8_isrForCrosstalkSources_946478_58.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946478/e0075896-d184-477b-bcba-176d6edeeac8_isrForCrosstalkSources_946478_58.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19290ad96687c95873cd935f8814260a with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c9679584-3d91-48cf-bc11-0a9355b28024_isrForCrosstalkSources_946369_41.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/c9679584-3d91-48cf-bc11-0a9355b28024_isrForCrosstalkSources_946369_41.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 84/9 running/pending blocks, and 281 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22970217758864 parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f0415515ccdb9d81fc568ab9d732950 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/f32d6e6e-cf11-412e-b706-8555cd17a5e9_isrForCrosstalkSources_946370_56.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/f32d6e6e-cf11-412e-b706-8555cd17a5e9_isrForCrosstalkSources_946370_56.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 280823f817462e7c662cb4d8bd74b455 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c313b4ac-fe99-41d7-8483-ccaa368260c1_isrForCrosstalkSources_946479_8.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/c313b4ac-fe99-41d7-8483-ccaa368260c1_isrForCrosstalkSources_946479_8.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cf4352d3b7afc1056164b5b9d78480c with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/ac76c08b-710a-41e3-8c45-72d02e4e3851_isrForCrosstalkSources_946370_32.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946370/ac76c08b-710a-41e3-8c45-72d02e4e3851_isrForCrosstalkSources_946370_32.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06f0dcad11da74b5a02068e39f910321 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ff7e6914-9461-4c95-ac79-ef59cf722abe_isrForCrosstalkSources_946371_19.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/ff7e6914-9461-4c95-ac79-ef59cf722abe_isrForCrosstalkSources_946371_19.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffe5a8de99578d9c84dd0bf15b2d89bf with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/28ee0843-4b69-43d7-9ba6-418e01d410e4_isrForCrosstalkSources_946369_35.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946369/28ee0843-4b69-43d7-9ba6-418e01d410e4_isrForCrosstalkSources_946369_35.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1ae523a27e838092ea5a204b2cab18 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/5a8feb52-3de2-4622-9556-4a0a4d1b9681_isrForCrosstalkSources_946371_24.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/5a8feb52-3de2-4622-9556-4a0a4d1b9681_isrForCrosstalkSources_946371_24.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e32e958896a4adf30a69598319288310 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/26ec77cd-4a58-4c4e-8553-4435fcf6fd1e_isrForCrosstalkSources_946479_33.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946479/26ec77cd-4a58-4c4e-8553-4435fcf6fd1e_isrForCrosstalkSources_946479_33.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f93ff075237946383f87491a0dff878a with result from task 354 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 372 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/b43671ee-5f4d-4875-9728-b554d24653be_isrForCrosstalkSources_946371_47.stdout parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/submit/DEEP/20201017/B1c/science#step0/20240313T231242Z/logs/isrForCrosstalkSources/946371/b43671ee-5f4d-4875-9728-b554d24653be_isrForCrosstalkSources_946371_47.stderr parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 84 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 542dca395547 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 542dca395547 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0763c77966d3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0763c77966d3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 11c343ab1912 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 11c343ab1912 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6dbe7576ff1f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6dbe7576ff1f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a6c19a2e5850 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a6c19a2e5850 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 88ade9b1bdf0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 88ade9b1bdf0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e01efbe7e391 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e01efbe7e391 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e59f777bb4d9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e59f777bb4d9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b3034f3529f0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b3034f3529f0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b4505ecb55d7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b4505ecb55d7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7690cab9eb05 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7690cab9eb05 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7622d6b1084f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7622d6b1084f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d0f5afb668c6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d0f5afb668c6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 03eb1668e530 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 03eb1668e530 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6777902156a7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6777902156a7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a7a6ca4e20a8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a7a6ca4e20a8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1153af9d7bfb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1153af9d7bfb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 57bcb8298c79 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 57bcb8298c79 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6a1ed07fcff2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6a1ed07fcff2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 97a937a29b4e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 97a937a29b4e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9ec0136bc0a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9ec0136bc0a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5bfddd40edf5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5bfddd40edf5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 909966a0dc93 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 909966a0dc93 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a863d11c242c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a863d11c242c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9595148fa5e8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9595148fa5e8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5f68162e688d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5f68162e688d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 84f4830bada2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 84f4830bada2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6283c20da129 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6283c20da129 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87bc20b54b94 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87bc20b54b94 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f438b6baf0c6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f438b6baf0c6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1497bd7e2dd3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1497bd7e2dd3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8b8a207d0fa9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8b8a207d0fa9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 19e81e085240 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 19e81e085240 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b30ad2c2d52b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b30ad2c2d52b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c10b9dc1c6a0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c10b9dc1c6a0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0dd34d6171c5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0dd34d6171c5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e1d75306655b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e1d75306655b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8b4f78a3e9f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8b4f78a3e9f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a3914c382d0f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a3914c382d0f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 57daf898b2bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 57daf898b2bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b57030c0f405 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b57030c0f405 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5ec597cd97f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5ec597cd97f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8bf3c2434325 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8bf3c2434325 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5f370a3632da parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5f370a3632da parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 257511095aa7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 257511095aa7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 479910141221 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 479910141221 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7860ee41009c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7860ee41009c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 296d0cefceb1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 296d0cefceb1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8314a8092699 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8314a8092699 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b231a5bf73d5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b231a5bf73d5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 45025a8cae8c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 45025a8cae8c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3e8bea13f24c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3e8bea13f24c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5a76ffdc15e8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5a76ffdc15e8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9ce46bbc1396 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9ce46bbc1396 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e4393cc811a7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e4393cc811a7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d6e3dec5e34 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d6e3dec5e34 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2f086c7ce4c4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2f086c7ce4c4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 317d108cb71e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 317d108cb71e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 53dc79e9e6b4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 53dc79e9e6b4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 11d132a7099c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 11d132a7099c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: af21c74c1308 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: af21c74c1308 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bff49f7637f5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bff49f7637f5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 232e350db1ff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 232e350db1ff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c6977bd02b60 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c6977bd02b60 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 509e0107f01a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 509e0107f01a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0911edc5b9b6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0911edc5b9b6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fb0141585cce parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fb0141585cce parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1dd4b341b686 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1dd4b341b686 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9cd1eea6b27b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9cd1eea6b27b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 36d425c4bb00 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 36d425c4bb00 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ad65cec3c7e9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ad65cec3c7e9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 52d6c72c2a79 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 52d6c72c2a79 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5a1f76f85778 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5a1f76f85778 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e44108bcf16a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e44108bcf16a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c0e6e3649f73 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c0e6e3649f73 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6ae699703a0a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6ae699703a0a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2afe3401df29 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2afe3401df29 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 91c9d153bf4a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 91c9d153bf4a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 620d94934cfd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 620d94934cfd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 380f9f075fa1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 380f9f075fa1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a70a794e1f4c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a70a794e1f4c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eb07e070cc15 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eb07e070cc15 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 107d3ae09b2a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 107d3ae09b2a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c981faf0d5dd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c981faf0d5dd 2024-03-13 23:14:43 proc_lsst.multi:146 [INFO] found job 17025306 in provider checkpoint proc_lsst.multi INFO: found job 17025306 in provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:201 [INFO] cancelling 17025306 on provider checkpoint proc_lsst.multi INFO: cancelling 17025306 on provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:146 [INFO] found job 17025308 in provider checkpoint proc_lsst.multi INFO: found job 17025308 in provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:201 [INFO] cancelling 17025308 on provider checkpoint proc_lsst.multi INFO: cancelling 17025308 on provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:146 [INFO] found job 17025296 in provider checkpoint proc_lsst.multi INFO: found job 17025296 in provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:201 [INFO] cancelling 17025296 on provider checkpoint proc_lsst.multi INFO: cancelling 17025296 on provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:146 [INFO] found job 17025290 in provider checkpoint proc_lsst.multi INFO: found job 17025290 in provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:201 [INFO] cancelling 17025290 on provider checkpoint proc_lsst.multi INFO: cancelling 17025290 on provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:146 [INFO] found job 17025295 in provider checkpoint proc_lsst.multi INFO: found job 17025295 in provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:201 [INFO] cancelling 17025295 on provider checkpoint proc_lsst.multi INFO: cancelling 17025295 on provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:146 [INFO] found job 17025294 in provider checkpoint proc_lsst.multi INFO: found job 17025294 in provider checkpoint 2024-03-13 23:14:43 proc_lsst.multi:201 [INFO] cancelling 17025294 on provider checkpoint proc_lsst.multi INFO: cancelling 17025294 on provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:146 [INFO] found job 17025281 in provider checkpoint proc_lsst.multi INFO: found job 17025281 in provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:201 [INFO] cancelling 17025281 on provider checkpoint proc_lsst.multi INFO: cancelling 17025281 on provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:146 [INFO] found job 17025284 in provider checkpoint proc_lsst.multi INFO: found job 17025284 in provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:201 [INFO] cancelling 17025284 on provider checkpoint proc_lsst.multi INFO: cancelling 17025284 on provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:146 [INFO] found job 17025283 in provider checkpoint proc_lsst.multi INFO: found job 17025283 in provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:201 [INFO] cancelling 17025283 on provider checkpoint proc_lsst.multi INFO: cancelling 17025283 on provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:146 [INFO] found job 17025303 in provider checkpoint proc_lsst.multi INFO: found job 17025303 in provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:201 [INFO] cancelling 17025303 on provider checkpoint proc_lsst.multi INFO: cancelling 17025303 on provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:146 [INFO] found job 17025304 in provider checkpoint proc_lsst.multi INFO: found job 17025304 in provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:201 [INFO] cancelling 17025304 on provider checkpoint proc_lsst.multi INFO: cancelling 17025304 on provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:146 [INFO] found job 17025305 in provider checkpoint proc_lsst.multi INFO: found job 17025305 in provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:201 [INFO] cancelling 17025305 on provider checkpoint proc_lsst.multi INFO: cancelling 17025305 on provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:146 [INFO] found job 17025302 in provider checkpoint proc_lsst.multi INFO: found job 17025302 in provider checkpoint 2024-03-13 23:14:44 proc_lsst.multi:201 [INFO] cancelling 17025302 on provider checkpoint proc_lsst.multi INFO: cancelling 17025302 on provider checkpoint 2024-03-13 23:14:45 proc_lsst.multi:146 [INFO] found job 17025289 in provider checkpoint proc_lsst.multi INFO: found job 17025289 in provider checkpoint 2024-03-13 23:14:45 proc_lsst.multi:201 [INFO] cancelling 17025289 on provider checkpoint proc_lsst.multi INFO: cancelling 17025289 on provider checkpoint 2024-03-13 23:14:45 proc_lsst.multi:146 [INFO] found job 17025307 in provider checkpoint proc_lsst.multi INFO: found job 17025307 in provider checkpoint 2024-03-13 23:14:45 proc_lsst.multi:201 [INFO] cancelling 17025307 on provider checkpoint proc_lsst.multi INFO: cancelling 17025307 on provider checkpoint 2024-03-13 23:14:45 proc_lsst.multi:146 [INFO] found job 17025282 in provider checkpoint proc_lsst.multi INFO: found job 17025282 in provider checkpoint 2024-03-13 23:14:45 proc_lsst.multi:201 [INFO] cancelling 17025282 on provider checkpoint proc_lsst.multi INFO: cancelling 17025282 on provider checkpoint 2024-03-13 23:14:45 proc_lsst.multi:146 [INFO] found job 17025263 in provider escience_a40 proc_lsst.multi INFO: found job 17025263 in provider escience_a40 2024-03-13 23:14:45 proc_lsst.multi:201 [INFO] cancelling 17025263 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025263 on provider escience_a40 2024-03-13 23:14:45 proc_lsst.multi:146 [INFO] found job 17025248 in provider astro proc_lsst.multi INFO: found job 17025248 in provider astro 2024-03-13 23:14:45 proc_lsst.multi:201 [INFO] cancelling 17025248 on provider astro proc_lsst.multi INFO: cancelling 17025248 on provider astro 2024-03-13 23:14:45 proc_lsst.multi:146 [INFO] found job 17025253 in provider astro proc_lsst.multi INFO: found job 17025253 in provider astro 2024-03-13 23:14:45 proc_lsst.multi:201 [INFO] cancelling 17025253 on provider astro proc_lsst.multi INFO: cancelling 17025253 on provider astro 2024-03-13 23:14:45 proc_lsst.multi:146 [INFO] found job 17025260 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025260 in provider escience_rtx6k 2024-03-13 23:14:45 proc_lsst.multi:201 [INFO] cancelling 17025260 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025260 on provider escience_rtx6k 2024-03-13 23:14:45 proc_lsst.multi:146 [INFO] found job 17025246 in provider astro proc_lsst.multi INFO: found job 17025246 in provider astro 2024-03-13 23:14:45 proc_lsst.multi:201 [INFO] cancelling 17025246 on provider astro proc_lsst.multi INFO: cancelling 17025246 on provider astro 2024-03-13 23:14:46 proc_lsst.multi:146 [INFO] found job 17025254 in provider astro proc_lsst.multi INFO: found job 17025254 in provider astro 2024-03-13 23:14:46 proc_lsst.multi:201 [INFO] cancelling 17025254 on provider astro proc_lsst.multi INFO: cancelling 17025254 on provider astro 2024-03-13 23:14:46 proc_lsst.multi:146 [INFO] found job 17025249 in provider astro proc_lsst.multi INFO: found job 17025249 in provider astro 2024-03-13 23:14:46 proc_lsst.multi:201 [INFO] cancelling 17025249 on provider astro proc_lsst.multi INFO: cancelling 17025249 on provider astro 2024-03-13 23:14:46 proc_lsst.multi:146 [INFO] found job 17025252 in provider astro proc_lsst.multi INFO: found job 17025252 in provider astro 2024-03-13 23:14:46 proc_lsst.multi:201 [INFO] cancelling 17025252 on provider astro proc_lsst.multi INFO: cancelling 17025252 on provider astro 2024-03-13 23:14:46 proc_lsst.multi:146 [INFO] found job 17025271 in provider escience_a40 proc_lsst.multi INFO: found job 17025271 in provider escience_a40 2024-03-13 23:14:46 proc_lsst.multi:201 [INFO] cancelling 17025271 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025271 on provider escience_a40 2024-03-13 23:14:46 proc_lsst.multi:146 [INFO] found job 17025247 in provider astro proc_lsst.multi INFO: found job 17025247 in provider astro 2024-03-13 23:14:46 proc_lsst.multi:201 [INFO] cancelling 17025247 on provider astro proc_lsst.multi INFO: cancelling 17025247 on provider astro 2024-03-13 23:14:46 proc_lsst.multi:146 [INFO] found job 17025251 in provider astro proc_lsst.multi INFO: found job 17025251 in provider astro 2024-03-13 23:14:46 proc_lsst.multi:201 [INFO] cancelling 17025251 on provider astro proc_lsst.multi INFO: cancelling 17025251 on provider astro 2024-03-13 23:14:46 proc_lsst.multi:146 [INFO] found job 17025250 in provider astro proc_lsst.multi INFO: found job 17025250 in provider astro 2024-03-13 23:14:46 proc_lsst.multi:201 [INFO] cancelling 17025250 on provider astro proc_lsst.multi INFO: cancelling 17025250 on provider astro 2024-03-13 23:14:46 proc_lsst.multi:146 [INFO] found job 17025255 in provider astro proc_lsst.multi INFO: found job 17025255 in provider astro 2024-03-13 23:14:46 proc_lsst.multi:201 [INFO] cancelling 17025255 on provider astro proc_lsst.multi INFO: cancelling 17025255 on provider astro 2024-03-13 23:14:47 proc_lsst.multi:146 [INFO] found job 17025274 in provider escience_a40 proc_lsst.multi INFO: found job 17025274 in provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:201 [INFO] cancelling 17025274 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025274 on provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:146 [INFO] found job 17025280 in provider escience_a40 proc_lsst.multi INFO: found job 17025280 in provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:201 [INFO] cancelling 17025280 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025280 on provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:146 [INFO] found job 17025257 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025257 in provider escience_rtx6k 2024-03-13 23:14:47 proc_lsst.multi:201 [INFO] cancelling 17025257 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025257 on provider escience_rtx6k 2024-03-13 23:14:47 proc_lsst.multi:146 [INFO] found job 17025276 in provider escience_a40 proc_lsst.multi INFO: found job 17025276 in provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:201 [INFO] cancelling 17025276 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025276 on provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:146 [INFO] found job 17025266 in provider escience_a40 proc_lsst.multi INFO: found job 17025266 in provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:201 [INFO] cancelling 17025266 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025266 on provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:146 [INFO] found job 17025267 in provider escience_a40 proc_lsst.multi INFO: found job 17025267 in provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:201 [INFO] cancelling 17025267 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025267 on provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:146 [INFO] found job 17025278 in provider escience_a40 proc_lsst.multi INFO: found job 17025278 in provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:201 [INFO] cancelling 17025278 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025278 on provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:146 [INFO] found job 17025270 in provider escience_a40 proc_lsst.multi INFO: found job 17025270 in provider escience_a40 2024-03-13 23:14:47 proc_lsst.multi:201 [INFO] cancelling 17025270 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025270 on provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:146 [INFO] found job 17025265 in provider escience_a40 proc_lsst.multi INFO: found job 17025265 in provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:201 [INFO] cancelling 17025265 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025265 on provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:146 [INFO] found job 17025269 in provider escience_a40 proc_lsst.multi INFO: found job 17025269 in provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:201 [INFO] cancelling 17025269 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025269 on provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:146 [INFO] found job 17025279 in provider escience_a40 proc_lsst.multi INFO: found job 17025279 in provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:201 [INFO] cancelling 17025279 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025279 on provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:146 [INFO] found job 17025264 in provider escience_a40 proc_lsst.multi INFO: found job 17025264 in provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:201 [INFO] cancelling 17025264 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025264 on provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:146 [INFO] found job 17025261 in provider escience_a40 proc_lsst.multi INFO: found job 17025261 in provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:201 [INFO] cancelling 17025261 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025261 on provider escience_a40 2024-03-13 23:14:48 proc_lsst.multi:146 [INFO] found job 17025258 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025258 in provider escience_rtx6k 2024-03-13 23:14:48 proc_lsst.multi:201 [INFO] cancelling 17025258 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025258 on provider escience_rtx6k 2024-03-13 23:14:48 proc_lsst.multi:146 [INFO] found job 17025256 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025256 in provider escience_rtx6k 2024-03-13 23:14:48 proc_lsst.multi:201 [INFO] cancelling 17025256 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025256 on provider escience_rtx6k 2024-03-13 23:14:49 proc_lsst.multi:146 [INFO] found job 17025275 in provider escience_a40 proc_lsst.multi INFO: found job 17025275 in provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:201 [INFO] cancelling 17025275 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025275 on provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:146 [INFO] found job 17025262 in provider escience_a40 proc_lsst.multi INFO: found job 17025262 in provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:201 [INFO] cancelling 17025262 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025262 on provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:146 [INFO] found job 17025268 in provider escience_a40 proc_lsst.multi INFO: found job 17025268 in provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:201 [INFO] cancelling 17025268 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025268 on provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:146 [INFO] found job 17025272 in provider escience_a40 proc_lsst.multi INFO: found job 17025272 in provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:201 [INFO] cancelling 17025272 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025272 on provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:146 [INFO] found job 17025273 in provider escience_a40 proc_lsst.multi INFO: found job 17025273 in provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:201 [INFO] cancelling 17025273 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025273 on provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:146 [INFO] found job 17025277 in provider escience_a40 proc_lsst.multi INFO: found job 17025277 in provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:201 [INFO] cancelling 17025277 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025277 on provider escience_a40 2024-03-13 23:14:49 proc_lsst.multi:146 [INFO] found job 17025259 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025259 in provider escience_rtx6k 2024-03-13 23:14:49 proc_lsst.multi:201 [INFO] cancelling 17025259 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025259 on provider escience_rtx6k 2024-03-13 23:14:49 proc_lsst.multi:146 [INFO] found job 17025288 in provider checkpoint proc_lsst.multi INFO: found job 17025288 in provider checkpoint 2024-03-13 23:14:49 proc_lsst.multi:201 [INFO] cancelling 17025288 on provider checkpoint proc_lsst.multi INFO: cancelling 17025288 on provider checkpoint 2024-03-13 23:14:50 proc_lsst.multi:146 [INFO] found job 17025287 in provider checkpoint proc_lsst.multi INFO: found job 17025287 in provider checkpoint 2024-03-13 23:14:50 proc_lsst.multi:201 [INFO] cancelling 17025287 on provider checkpoint proc_lsst.multi INFO: cancelling 17025287 on provider checkpoint 2024-03-13 23:14:50 proc_lsst.multi:146 [INFO] found job 20175 in provider local proc_lsst.multi INFO: found job 20175 in provider local 2024-03-13 23:14:50 proc_lsst.multi:201 [INFO] cancelling 20175 on provider local proc_lsst.multi INFO: cancelling 20175 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 20175 2024-03-13 23:14:50 proc_lsst.multi:146 [INFO] found job 17025222 in provider astro proc_lsst.multi INFO: found job 17025222 in provider astro 2024-03-13 23:14:50 proc_lsst.multi:201 [INFO] cancelling 17025222 on provider astro proc_lsst.multi INFO: cancelling 17025222 on provider astro 2024-03-13 23:14:50 proc_lsst.multi:146 [INFO] found job 17025208 in provider astro proc_lsst.multi INFO: found job 17025208 in provider astro 2024-03-13 23:14:50 proc_lsst.multi:201 [INFO] cancelling 17025208 on provider astro proc_lsst.multi INFO: cancelling 17025208 on provider astro 2024-03-13 23:14:50 proc_lsst.multi:146 [INFO] found job 17025286 in provider checkpoint proc_lsst.multi INFO: found job 17025286 in provider checkpoint 2024-03-13 23:14:50 proc_lsst.multi:201 [INFO] cancelling 17025286 on provider checkpoint proc_lsst.multi INFO: cancelling 17025286 on provider checkpoint 2024-03-13 23:14:50 proc_lsst.multi:146 [INFO] found job 17025207 in provider astro proc_lsst.multi INFO: found job 17025207 in provider astro 2024-03-13 23:14:50 proc_lsst.multi:201 [INFO] cancelling 17025207 on provider astro proc_lsst.multi INFO: cancelling 17025207 on provider astro 2024-03-13 23:14:50 proc_lsst.multi:146 [INFO] found job 17025243 in provider astro proc_lsst.multi INFO: found job 17025243 in provider astro 2024-03-13 23:14:50 proc_lsst.multi:201 [INFO] cancelling 17025243 on provider astro proc_lsst.multi INFO: cancelling 17025243 on provider astro 2024-03-13 23:14:50 proc_lsst.multi:146 [INFO] found job 17025245 in provider astro proc_lsst.multi INFO: found job 17025245 in provider astro 2024-03-13 23:14:50 proc_lsst.multi:201 [INFO] cancelling 17025245 on provider astro proc_lsst.multi INFO: cancelling 17025245 on provider astro 2024-03-13 23:14:51 proc_lsst.multi:146 [INFO] found job 17025225 in provider astro proc_lsst.multi INFO: found job 17025225 in provider astro 2024-03-13 23:14:51 proc_lsst.multi:201 [INFO] cancelling 17025225 on provider astro proc_lsst.multi INFO: cancelling 17025225 on provider astro 2024-03-13 23:14:51 proc_lsst.multi:146 [INFO] found job 17025224 in provider astro proc_lsst.multi INFO: found job 17025224 in provider astro 2024-03-13 23:14:51 proc_lsst.multi:201 [INFO] cancelling 17025224 on provider astro proc_lsst.multi INFO: cancelling 17025224 on provider astro 2024-03-13 23:14:51 proc_lsst.multi:146 [INFO] found job 17025220 in provider astro proc_lsst.multi INFO: found job 17025220 in provider astro 2024-03-13 23:14:51 proc_lsst.multi:201 [INFO] cancelling 17025220 on provider astro proc_lsst.multi INFO: cancelling 17025220 on provider astro 2024-03-13 23:14:51 proc_lsst.multi:146 [INFO] found job 17025244 in provider astro proc_lsst.multi INFO: found job 17025244 in provider astro 2024-03-13 23:14:51 proc_lsst.multi:201 [INFO] cancelling 17025244 on provider astro proc_lsst.multi INFO: cancelling 17025244 on provider astro 2024-03-13 23:14:51 proc_lsst.multi:146 [INFO] found job 17025227 in provider astro proc_lsst.multi INFO: found job 17025227 in provider astro 2024-03-13 23:14:51 proc_lsst.multi:201 [INFO] cancelling 17025227 on provider astro proc_lsst.multi INFO: cancelling 17025227 on provider astro 2024-03-13 23:14:51 proc_lsst.multi:146 [INFO] found job 17025223 in provider astro proc_lsst.multi INFO: found job 17025223 in provider astro 2024-03-13 23:14:51 proc_lsst.multi:201 [INFO] cancelling 17025223 on provider astro proc_lsst.multi INFO: cancelling 17025223 on provider astro 2024-03-13 23:14:51 proc_lsst.multi:146 [INFO] found job 17025221 in provider astro proc_lsst.multi INFO: found job 17025221 in provider astro 2024-03-13 23:14:51 proc_lsst.multi:201 [INFO] cancelling 17025221 on provider astro proc_lsst.multi INFO: cancelling 17025221 on provider astro 2024-03-13 23:14:52 proc_lsst.multi:146 [INFO] found job 17025240 in provider astro proc_lsst.multi INFO: found job 17025240 in provider astro 2024-03-13 23:14:52 proc_lsst.multi:201 [INFO] cancelling 17025240 on provider astro proc_lsst.multi INFO: cancelling 17025240 on provider astro 2024-03-13 23:14:52 proc_lsst.multi:146 [INFO] found job 17025239 in provider astro proc_lsst.multi INFO: found job 17025239 in provider astro 2024-03-13 23:14:52 proc_lsst.multi:201 [INFO] cancelling 17025239 on provider astro proc_lsst.multi INFO: cancelling 17025239 on provider astro 2024-03-13 23:14:52 proc_lsst.multi:146 [INFO] found job 17025226 in provider astro proc_lsst.multi INFO: found job 17025226 in provider astro 2024-03-13 23:14:52 proc_lsst.multi:201 [INFO] cancelling 17025226 on provider astro proc_lsst.multi INFO: cancelling 17025226 on provider astro 2024-03-13 23:14:52 proc_lsst.multi:146 [INFO] found job 17025238 in provider astro proc_lsst.multi INFO: found job 17025238 in provider astro 2024-03-13 23:14:52 proc_lsst.multi:201 [INFO] cancelling 17025238 on provider astro proc_lsst.multi INFO: cancelling 17025238 on provider astro 2024-03-13 23:14:52 proc_lsst.multi:146 [INFO] found job 17025219 in provider astro proc_lsst.multi INFO: found job 17025219 in provider astro 2024-03-13 23:14:52 proc_lsst.multi:201 [INFO] cancelling 17025219 on provider astro proc_lsst.multi INFO: cancelling 17025219 on provider astro 2024-03-13 23:14:52 proc_lsst.multi:146 [INFO] found job 17025241 in provider astro proc_lsst.multi INFO: found job 17025241 in provider astro 2024-03-13 23:14:52 proc_lsst.multi:201 [INFO] cancelling 17025241 on provider astro proc_lsst.multi INFO: cancelling 17025241 on provider astro 2024-03-13 23:14:52 proc_lsst.multi:146 [INFO] found job 17025242 in provider astro proc_lsst.multi INFO: found job 17025242 in provider astro 2024-03-13 23:14:52 proc_lsst.multi:201 [INFO] cancelling 17025242 on provider astro proc_lsst.multi INFO: cancelling 17025242 on provider astro 2024-03-13 23:14:53 proc_lsst.multi:146 [INFO] found job 17025228 in provider astro proc_lsst.multi INFO: found job 17025228 in provider astro 2024-03-13 23:14:53 proc_lsst.multi:201 [INFO] cancelling 17025228 on provider astro proc_lsst.multi INFO: cancelling 17025228 on provider astro 2024-03-13 23:14:53 proc_lsst.multi:146 [INFO] found job 17025285 in provider checkpoint proc_lsst.multi INFO: found job 17025285 in provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:201 [INFO] cancelling 17025285 on provider checkpoint proc_lsst.multi INFO: cancelling 17025285 on provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:146 [INFO] found job 17025297 in provider checkpoint proc_lsst.multi INFO: found job 17025297 in provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:201 [INFO] cancelling 17025297 on provider checkpoint proc_lsst.multi INFO: cancelling 17025297 on provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:146 [INFO] found job 17025298 in provider checkpoint proc_lsst.multi INFO: found job 17025298 in provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:201 [INFO] cancelling 17025298 on provider checkpoint proc_lsst.multi INFO: cancelling 17025298 on provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:146 [INFO] found job 17025299 in provider checkpoint proc_lsst.multi INFO: found job 17025299 in provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:201 [INFO] cancelling 17025299 on provider checkpoint proc_lsst.multi INFO: cancelling 17025299 on provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:146 [INFO] found job 17025301 in provider checkpoint proc_lsst.multi INFO: found job 17025301 in provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:201 [INFO] cancelling 17025301 on provider checkpoint proc_lsst.multi INFO: cancelling 17025301 on provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:146 [INFO] found job 17025300 in provider checkpoint proc_lsst.multi INFO: found job 17025300 in provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:201 [INFO] cancelling 17025300 on provider checkpoint proc_lsst.multi INFO: cancelling 17025300 on provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:146 [INFO] found job 17025292 in provider checkpoint proc_lsst.multi INFO: found job 17025292 in provider checkpoint 2024-03-13 23:14:53 proc_lsst.multi:201 [INFO] cancelling 17025292 on provider checkpoint proc_lsst.multi INFO: cancelling 17025292 on provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:146 [INFO] found job 17025293 in provider checkpoint proc_lsst.multi INFO: found job 17025293 in provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:201 [INFO] cancelling 17025293 on provider checkpoint proc_lsst.multi INFO: cancelling 17025293 on provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:146 [INFO] found job 17025291 in provider checkpoint proc_lsst.multi INFO: found job 17025291 in provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:201 [INFO] cancelling 17025291 on provider checkpoint proc_lsst.multi INFO: cancelling 17025291 on provider checkpoint parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-13 23:14:54 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-13 23:14:54 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17025281', '17025282', '17025283', '17025284', '17025285', '17025286', '17025287', '17025288', '17025289', '17025290', '17025291', '17025292', '17025293', '17025294', '17025295', '17025296', '17025297', '17025298', '17025299', '17025300', '17025301', '17025302', '17025303', '17025304', '17025305', '17025306', '17025307', '17025308', '17025309', '17025310', '17025311', '17025312', '17025313', '17025314', '17025315', '17025316', '17025317', '17025207', '17025208', '17025219', '17025220', '17025221', '17025222', '17025223', '17025224', '17025225', '17025226', '17025227', '17025228', '17025238', '17025239', '17025240', '17025241', '17025242', '17025243', '17025244', '17025245', '17025246', '17025247', '17025248', '17025249', '17025250', '17025251', '17025252', '17025253', '17025254', '17025255', '17025261', '17025262', '17025263', '17025264', '17025265', '17025266', '17025267', '17025268', '17025269', '17025270', '17025271', '17025272', '17025273', '17025274', '17025275', '17025276', '17025277', '17025278', '17025279', '17025280', '17025256', '17025257', '17025258', '17025259', '17025260', '20175'] proc_lsst.multi INFO: new jobs since last cancel ['17025281', '17025282', '17025283', '17025284', '17025285', '17025286', '17025287', '17025288', '17025289', '17025290', '17025291', '17025292', '17025293', '17025294', '17025295', '17025296', '17025297', '17025298', '17025299', '17025300', '17025301', '17025302', '17025303', '17025304', '17025305', '17025306', '17025307', '17025308', '17025309', '17025310', '17025311', '17025312', '17025313', '17025314', '17025315', '17025316', '17025317', '17025207', '17025208', '17025219', '17025220', '17025221', '17025222', '17025223', '17025224', '17025225', '17025226', '17025227', '17025228', '17025238', '17025239', '17025240', '17025241', '17025242', '17025243', '17025244', '17025245', '17025246', '17025247', '17025248', '17025249', '17025250', '17025251', '17025252', '17025253', '17025254', '17025255', '17025261', '17025262', '17025263', '17025264', '17025265', '17025266', '17025267', '17025268', '17025269', '17025270', '17025271', '17025272', '17025273', '17025274', '17025275', '17025276', '17025277', '17025278', '17025279', '17025280', '17025256', '17025257', '17025258', '17025259', '17025260', '20175'] 2024-03-13 23:14:54 proc_lsst.multi:146 [INFO] found job 17025281 in provider checkpoint proc_lsst.multi INFO: found job 17025281 in provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:201 [INFO] cancelling 17025281 on provider checkpoint proc_lsst.multi INFO: cancelling 17025281 on provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:146 [INFO] found job 17025282 in provider checkpoint proc_lsst.multi INFO: found job 17025282 in provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:201 [INFO] cancelling 17025282 on provider checkpoint proc_lsst.multi INFO: cancelling 17025282 on provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:146 [INFO] found job 17025283 in provider checkpoint proc_lsst.multi INFO: found job 17025283 in provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:201 [INFO] cancelling 17025283 on provider checkpoint proc_lsst.multi INFO: cancelling 17025283 on provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:146 [INFO] found job 17025284 in provider checkpoint proc_lsst.multi INFO: found job 17025284 in provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:201 [INFO] cancelling 17025284 on provider checkpoint proc_lsst.multi INFO: cancelling 17025284 on provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:146 [INFO] found job 17025285 in provider checkpoint proc_lsst.multi INFO: found job 17025285 in provider checkpoint 2024-03-13 23:14:54 proc_lsst.multi:201 [INFO] cancelling 17025285 on provider checkpoint proc_lsst.multi INFO: cancelling 17025285 on provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:146 [INFO] found job 17025286 in provider checkpoint proc_lsst.multi INFO: found job 17025286 in provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:201 [INFO] cancelling 17025286 on provider checkpoint proc_lsst.multi INFO: cancelling 17025286 on provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:146 [INFO] found job 17025287 in provider checkpoint proc_lsst.multi INFO: found job 17025287 in provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:201 [INFO] cancelling 17025287 on provider checkpoint proc_lsst.multi INFO: cancelling 17025287 on provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:146 [INFO] found job 17025288 in provider checkpoint proc_lsst.multi INFO: found job 17025288 in provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:201 [INFO] cancelling 17025288 on provider checkpoint proc_lsst.multi INFO: cancelling 17025288 on provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:146 [INFO] found job 17025289 in provider checkpoint proc_lsst.multi INFO: found job 17025289 in provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:201 [INFO] cancelling 17025289 on provider checkpoint proc_lsst.multi INFO: cancelling 17025289 on provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:146 [INFO] found job 17025290 in provider checkpoint proc_lsst.multi INFO: found job 17025290 in provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:201 [INFO] cancelling 17025290 on provider checkpoint proc_lsst.multi INFO: cancelling 17025290 on provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:146 [INFO] found job 17025291 in provider checkpoint proc_lsst.multi INFO: found job 17025291 in provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:201 [INFO] cancelling 17025291 on provider checkpoint proc_lsst.multi INFO: cancelling 17025291 on provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:146 [INFO] found job 17025292 in provider checkpoint proc_lsst.multi INFO: found job 17025292 in provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:201 [INFO] cancelling 17025292 on provider checkpoint proc_lsst.multi INFO: cancelling 17025292 on provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:146 [INFO] found job 17025293 in provider checkpoint proc_lsst.multi INFO: found job 17025293 in provider checkpoint 2024-03-13 23:14:55 proc_lsst.multi:201 [INFO] cancelling 17025293 on provider checkpoint proc_lsst.multi INFO: cancelling 17025293 on provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:146 [INFO] found job 17025294 in provider checkpoint proc_lsst.multi INFO: found job 17025294 in provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:201 [INFO] cancelling 17025294 on provider checkpoint proc_lsst.multi INFO: cancelling 17025294 on provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:146 [INFO] found job 17025295 in provider checkpoint proc_lsst.multi INFO: found job 17025295 in provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:201 [INFO] cancelling 17025295 on provider checkpoint proc_lsst.multi INFO: cancelling 17025295 on provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:146 [INFO] found job 17025296 in provider checkpoint proc_lsst.multi INFO: found job 17025296 in provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:201 [INFO] cancelling 17025296 on provider checkpoint proc_lsst.multi INFO: cancelling 17025296 on provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:146 [INFO] found job 17025297 in provider checkpoint proc_lsst.multi INFO: found job 17025297 in provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:201 [INFO] cancelling 17025297 on provider checkpoint proc_lsst.multi INFO: cancelling 17025297 on provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:146 [INFO] found job 17025298 in provider checkpoint proc_lsst.multi INFO: found job 17025298 in provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:201 [INFO] cancelling 17025298 on provider checkpoint proc_lsst.multi INFO: cancelling 17025298 on provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:146 [INFO] found job 17025299 in provider checkpoint proc_lsst.multi INFO: found job 17025299 in provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:201 [INFO] cancelling 17025299 on provider checkpoint proc_lsst.multi INFO: cancelling 17025299 on provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:146 [INFO] found job 17025300 in provider checkpoint proc_lsst.multi INFO: found job 17025300 in provider checkpoint 2024-03-13 23:14:56 proc_lsst.multi:201 [INFO] cancelling 17025300 on provider checkpoint proc_lsst.multi INFO: cancelling 17025300 on provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:146 [INFO] found job 17025301 in provider checkpoint proc_lsst.multi INFO: found job 17025301 in provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:201 [INFO] cancelling 17025301 on provider checkpoint proc_lsst.multi INFO: cancelling 17025301 on provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:146 [INFO] found job 17025302 in provider checkpoint proc_lsst.multi INFO: found job 17025302 in provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:201 [INFO] cancelling 17025302 on provider checkpoint proc_lsst.multi INFO: cancelling 17025302 on provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:146 [INFO] found job 17025303 in provider checkpoint proc_lsst.multi INFO: found job 17025303 in provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:201 [INFO] cancelling 17025303 on provider checkpoint proc_lsst.multi INFO: cancelling 17025303 on provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:146 [INFO] found job 17025304 in provider checkpoint proc_lsst.multi INFO: found job 17025304 in provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:201 [INFO] cancelling 17025304 on provider checkpoint proc_lsst.multi INFO: cancelling 17025304 on provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:146 [INFO] found job 17025305 in provider checkpoint proc_lsst.multi INFO: found job 17025305 in provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:201 [INFO] cancelling 17025305 on provider checkpoint proc_lsst.multi INFO: cancelling 17025305 on provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:146 [INFO] found job 17025306 in provider checkpoint proc_lsst.multi INFO: found job 17025306 in provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:201 [INFO] cancelling 17025306 on provider checkpoint proc_lsst.multi INFO: cancelling 17025306 on provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:146 [INFO] found job 17025307 in provider checkpoint proc_lsst.multi INFO: found job 17025307 in provider checkpoint 2024-03-13 23:14:57 proc_lsst.multi:201 [INFO] cancelling 17025307 on provider checkpoint proc_lsst.multi INFO: cancelling 17025307 on provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:146 [INFO] found job 17025308 in provider checkpoint proc_lsst.multi INFO: found job 17025308 in provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:201 [INFO] cancelling 17025308 on provider checkpoint proc_lsst.multi INFO: cancelling 17025308 on provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:146 [INFO] found job 17025309 in provider checkpoint proc_lsst.multi INFO: found job 17025309 in provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:201 [INFO] cancelling 17025309 on provider checkpoint proc_lsst.multi INFO: cancelling 17025309 on provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:146 [INFO] found job 17025310 in provider checkpoint proc_lsst.multi INFO: found job 17025310 in provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:201 [INFO] cancelling 17025310 on provider checkpoint proc_lsst.multi INFO: cancelling 17025310 on provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:146 [INFO] found job 17025311 in provider checkpoint proc_lsst.multi INFO: found job 17025311 in provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:201 [INFO] cancelling 17025311 on provider checkpoint proc_lsst.multi INFO: cancelling 17025311 on provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:146 [INFO] found job 17025312 in provider checkpoint proc_lsst.multi INFO: found job 17025312 in provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:201 [INFO] cancelling 17025312 on provider checkpoint proc_lsst.multi INFO: cancelling 17025312 on provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:146 [INFO] found job 17025313 in provider checkpoint proc_lsst.multi INFO: found job 17025313 in provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:201 [INFO] cancelling 17025313 on provider checkpoint proc_lsst.multi INFO: cancelling 17025313 on provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:146 [INFO] found job 17025314 in provider checkpoint proc_lsst.multi INFO: found job 17025314 in provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:201 [INFO] cancelling 17025314 on provider checkpoint proc_lsst.multi INFO: cancelling 17025314 on provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:146 [INFO] found job 17025315 in provider checkpoint proc_lsst.multi INFO: found job 17025315 in provider checkpoint 2024-03-13 23:14:58 proc_lsst.multi:201 [INFO] cancelling 17025315 on provider checkpoint proc_lsst.multi INFO: cancelling 17025315 on provider checkpoint 2024-03-13 23:14:59 proc_lsst.multi:146 [INFO] found job 17025316 in provider checkpoint proc_lsst.multi INFO: found job 17025316 in provider checkpoint 2024-03-13 23:14:59 proc_lsst.multi:201 [INFO] cancelling 17025316 on provider checkpoint proc_lsst.multi INFO: cancelling 17025316 on provider checkpoint 2024-03-13 23:14:59 proc_lsst.multi:146 [INFO] found job 17025317 in provider checkpoint proc_lsst.multi INFO: found job 17025317 in provider checkpoint 2024-03-13 23:14:59 proc_lsst.multi:201 [INFO] cancelling 17025317 on provider checkpoint proc_lsst.multi INFO: cancelling 17025317 on provider checkpoint 2024-03-13 23:14:59 proc_lsst.multi:146 [INFO] found job 17025207 in provider astro proc_lsst.multi INFO: found job 17025207 in provider astro 2024-03-13 23:14:59 proc_lsst.multi:201 [INFO] cancelling 17025207 on provider astro proc_lsst.multi INFO: cancelling 17025207 on provider astro 2024-03-13 23:14:59 proc_lsst.multi:146 [INFO] found job 17025208 in provider astro proc_lsst.multi INFO: found job 17025208 in provider astro 2024-03-13 23:14:59 proc_lsst.multi:201 [INFO] cancelling 17025208 on provider astro proc_lsst.multi INFO: cancelling 17025208 on provider astro 2024-03-13 23:14:59 proc_lsst.multi:146 [INFO] found job 17025219 in provider astro proc_lsst.multi INFO: found job 17025219 in provider astro 2024-03-13 23:14:59 proc_lsst.multi:201 [INFO] cancelling 17025219 on provider astro proc_lsst.multi INFO: cancelling 17025219 on provider astro 2024-03-13 23:14:59 proc_lsst.multi:146 [INFO] found job 17025220 in provider astro proc_lsst.multi INFO: found job 17025220 in provider astro 2024-03-13 23:14:59 proc_lsst.multi:201 [INFO] cancelling 17025220 on provider astro proc_lsst.multi INFO: cancelling 17025220 on provider astro 2024-03-13 23:14:59 proc_lsst.multi:146 [INFO] found job 17025221 in provider astro proc_lsst.multi INFO: found job 17025221 in provider astro 2024-03-13 23:14:59 proc_lsst.multi:201 [INFO] cancelling 17025221 on provider astro proc_lsst.multi INFO: cancelling 17025221 on provider astro 2024-03-13 23:14:59 proc_lsst.multi:146 [INFO] found job 17025222 in provider astro proc_lsst.multi INFO: found job 17025222 in provider astro 2024-03-13 23:14:59 proc_lsst.multi:201 [INFO] cancelling 17025222 on provider astro proc_lsst.multi INFO: cancelling 17025222 on provider astro 2024-03-13 23:15:00 proc_lsst.multi:146 [INFO] found job 17025223 in provider astro proc_lsst.multi INFO: found job 17025223 in provider astro 2024-03-13 23:15:00 proc_lsst.multi:201 [INFO] cancelling 17025223 on provider astro proc_lsst.multi INFO: cancelling 17025223 on provider astro 2024-03-13 23:15:00 proc_lsst.multi:146 [INFO] found job 17025224 in provider astro proc_lsst.multi INFO: found job 17025224 in provider astro 2024-03-13 23:15:00 proc_lsst.multi:201 [INFO] cancelling 17025224 on provider astro proc_lsst.multi INFO: cancelling 17025224 on provider astro 2024-03-13 23:15:00 proc_lsst.multi:146 [INFO] found job 17025225 in provider astro proc_lsst.multi INFO: found job 17025225 in provider astro 2024-03-13 23:15:00 proc_lsst.multi:201 [INFO] cancelling 17025225 on provider astro proc_lsst.multi INFO: cancelling 17025225 on provider astro 2024-03-13 23:15:00 proc_lsst.multi:146 [INFO] found job 17025226 in provider astro proc_lsst.multi INFO: found job 17025226 in provider astro 2024-03-13 23:15:00 proc_lsst.multi:201 [INFO] cancelling 17025226 on provider astro proc_lsst.multi INFO: cancelling 17025226 on provider astro 2024-03-13 23:15:00 proc_lsst.multi:146 [INFO] found job 17025227 in provider astro proc_lsst.multi INFO: found job 17025227 in provider astro 2024-03-13 23:15:00 proc_lsst.multi:201 [INFO] cancelling 17025227 on provider astro proc_lsst.multi INFO: cancelling 17025227 on provider astro 2024-03-13 23:15:00 proc_lsst.multi:146 [INFO] found job 17025228 in provider astro proc_lsst.multi INFO: found job 17025228 in provider astro 2024-03-13 23:15:00 proc_lsst.multi:201 [INFO] cancelling 17025228 on provider astro proc_lsst.multi INFO: cancelling 17025228 on provider astro 2024-03-13 23:15:00 proc_lsst.multi:146 [INFO] found job 17025238 in provider astro proc_lsst.multi INFO: found job 17025238 in provider astro 2024-03-13 23:15:00 proc_lsst.multi:201 [INFO] cancelling 17025238 on provider astro proc_lsst.multi INFO: cancelling 17025238 on provider astro 2024-03-13 23:15:01 proc_lsst.multi:146 [INFO] found job 17025239 in provider astro proc_lsst.multi INFO: found job 17025239 in provider astro 2024-03-13 23:15:01 proc_lsst.multi:201 [INFO] cancelling 17025239 on provider astro proc_lsst.multi INFO: cancelling 17025239 on provider astro 2024-03-13 23:15:01 proc_lsst.multi:146 [INFO] found job 17025240 in provider astro proc_lsst.multi INFO: found job 17025240 in provider astro 2024-03-13 23:15:01 proc_lsst.multi:201 [INFO] cancelling 17025240 on provider astro proc_lsst.multi INFO: cancelling 17025240 on provider astro 2024-03-13 23:15:01 proc_lsst.multi:146 [INFO] found job 17025241 in provider astro proc_lsst.multi INFO: found job 17025241 in provider astro 2024-03-13 23:15:01 proc_lsst.multi:201 [INFO] cancelling 17025241 on provider astro proc_lsst.multi INFO: cancelling 17025241 on provider astro 2024-03-13 23:15:01 proc_lsst.multi:146 [INFO] found job 17025242 in provider astro proc_lsst.multi INFO: found job 17025242 in provider astro 2024-03-13 23:15:01 proc_lsst.multi:201 [INFO] cancelling 17025242 on provider astro proc_lsst.multi INFO: cancelling 17025242 on provider astro 2024-03-13 23:15:01 proc_lsst.multi:146 [INFO] found job 17025243 in provider astro proc_lsst.multi INFO: found job 17025243 in provider astro 2024-03-13 23:15:01 proc_lsst.multi:201 [INFO] cancelling 17025243 on provider astro proc_lsst.multi INFO: cancelling 17025243 on provider astro 2024-03-13 23:15:01 proc_lsst.multi:146 [INFO] found job 17025244 in provider astro proc_lsst.multi INFO: found job 17025244 in provider astro 2024-03-13 23:15:01 proc_lsst.multi:201 [INFO] cancelling 17025244 on provider astro proc_lsst.multi INFO: cancelling 17025244 on provider astro 2024-03-13 23:15:01 proc_lsst.multi:146 [INFO] found job 17025245 in provider astro proc_lsst.multi INFO: found job 17025245 in provider astro 2024-03-13 23:15:01 proc_lsst.multi:201 [INFO] cancelling 17025245 on provider astro proc_lsst.multi INFO: cancelling 17025245 on provider astro 2024-03-13 23:15:01 proc_lsst.multi:146 [INFO] found job 17025246 in provider astro proc_lsst.multi INFO: found job 17025246 in provider astro 2024-03-13 23:15:01 proc_lsst.multi:201 [INFO] cancelling 17025246 on provider astro proc_lsst.multi INFO: cancelling 17025246 on provider astro 2024-03-13 23:15:02 proc_lsst.multi:146 [INFO] found job 17025247 in provider astro proc_lsst.multi INFO: found job 17025247 in provider astro 2024-03-13 23:15:02 proc_lsst.multi:201 [INFO] cancelling 17025247 on provider astro proc_lsst.multi INFO: cancelling 17025247 on provider astro 2024-03-13 23:15:02 proc_lsst.multi:146 [INFO] found job 17025248 in provider astro proc_lsst.multi INFO: found job 17025248 in provider astro 2024-03-13 23:15:02 proc_lsst.multi:201 [INFO] cancelling 17025248 on provider astro proc_lsst.multi INFO: cancelling 17025248 on provider astro 2024-03-13 23:15:02 proc_lsst.multi:146 [INFO] found job 17025249 in provider astro proc_lsst.multi INFO: found job 17025249 in provider astro 2024-03-13 23:15:02 proc_lsst.multi:201 [INFO] cancelling 17025249 on provider astro proc_lsst.multi INFO: cancelling 17025249 on provider astro 2024-03-13 23:15:02 proc_lsst.multi:146 [INFO] found job 17025250 in provider astro proc_lsst.multi INFO: found job 17025250 in provider astro 2024-03-13 23:15:02 proc_lsst.multi:201 [INFO] cancelling 17025250 on provider astro proc_lsst.multi INFO: cancelling 17025250 on provider astro 2024-03-13 23:15:02 proc_lsst.multi:146 [INFO] found job 17025251 in provider astro proc_lsst.multi INFO: found job 17025251 in provider astro 2024-03-13 23:15:02 proc_lsst.multi:201 [INFO] cancelling 17025251 on provider astro proc_lsst.multi INFO: cancelling 17025251 on provider astro 2024-03-13 23:15:02 proc_lsst.multi:146 [INFO] found job 17025252 in provider astro proc_lsst.multi INFO: found job 17025252 in provider astro 2024-03-13 23:15:02 proc_lsst.multi:201 [INFO] cancelling 17025252 on provider astro proc_lsst.multi INFO: cancelling 17025252 on provider astro 2024-03-13 23:15:02 proc_lsst.multi:146 [INFO] found job 17025253 in provider astro proc_lsst.multi INFO: found job 17025253 in provider astro 2024-03-13 23:15:02 proc_lsst.multi:201 [INFO] cancelling 17025253 on provider astro proc_lsst.multi INFO: cancelling 17025253 on provider astro 2024-03-13 23:15:02 proc_lsst.multi:146 [INFO] found job 17025254 in provider astro proc_lsst.multi INFO: found job 17025254 in provider astro 2024-03-13 23:15:02 proc_lsst.multi:201 [INFO] cancelling 17025254 on provider astro proc_lsst.multi INFO: cancelling 17025254 on provider astro 2024-03-13 23:15:03 proc_lsst.multi:146 [INFO] found job 17025255 in provider astro proc_lsst.multi INFO: found job 17025255 in provider astro 2024-03-13 23:15:03 proc_lsst.multi:201 [INFO] cancelling 17025255 on provider astro proc_lsst.multi INFO: cancelling 17025255 on provider astro 2024-03-13 23:15:03 proc_lsst.multi:146 [INFO] found job 17025261 in provider escience_a40 proc_lsst.multi INFO: found job 17025261 in provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:201 [INFO] cancelling 17025261 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025261 on provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:146 [INFO] found job 17025262 in provider escience_a40 proc_lsst.multi INFO: found job 17025262 in provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:201 [INFO] cancelling 17025262 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025262 on provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:146 [INFO] found job 17025263 in provider escience_a40 proc_lsst.multi INFO: found job 17025263 in provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:201 [INFO] cancelling 17025263 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025263 on provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:146 [INFO] found job 17025264 in provider escience_a40 proc_lsst.multi INFO: found job 17025264 in provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:201 [INFO] cancelling 17025264 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025264 on provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:146 [INFO] found job 17025265 in provider escience_a40 proc_lsst.multi INFO: found job 17025265 in provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:201 [INFO] cancelling 17025265 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025265 on provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:146 [INFO] found job 17025266 in provider escience_a40 proc_lsst.multi INFO: found job 17025266 in provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:201 [INFO] cancelling 17025266 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025266 on provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:146 [INFO] found job 17025267 in provider escience_a40 proc_lsst.multi INFO: found job 17025267 in provider escience_a40 2024-03-13 23:15:03 proc_lsst.multi:201 [INFO] cancelling 17025267 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025267 on provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:146 [INFO] found job 17025268 in provider escience_a40 proc_lsst.multi INFO: found job 17025268 in provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:201 [INFO] cancelling 17025268 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025268 on provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:146 [INFO] found job 17025269 in provider escience_a40 proc_lsst.multi INFO: found job 17025269 in provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:201 [INFO] cancelling 17025269 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025269 on provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:146 [INFO] found job 17025270 in provider escience_a40 proc_lsst.multi INFO: found job 17025270 in provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:201 [INFO] cancelling 17025270 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025270 on provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:146 [INFO] found job 17025271 in provider escience_a40 proc_lsst.multi INFO: found job 17025271 in provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:201 [INFO] cancelling 17025271 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025271 on provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:146 [INFO] found job 17025272 in provider escience_a40 proc_lsst.multi INFO: found job 17025272 in provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:201 [INFO] cancelling 17025272 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025272 on provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:146 [INFO] found job 17025273 in provider escience_a40 proc_lsst.multi INFO: found job 17025273 in provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:201 [INFO] cancelling 17025273 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025273 on provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:146 [INFO] found job 17025274 in provider escience_a40 proc_lsst.multi INFO: found job 17025274 in provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:201 [INFO] cancelling 17025274 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025274 on provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:146 [INFO] found job 17025275 in provider escience_a40 proc_lsst.multi INFO: found job 17025275 in provider escience_a40 2024-03-13 23:15:04 proc_lsst.multi:201 [INFO] cancelling 17025275 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025275 on provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:146 [INFO] found job 17025276 in provider escience_a40 proc_lsst.multi INFO: found job 17025276 in provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:201 [INFO] cancelling 17025276 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025276 on provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:146 [INFO] found job 17025277 in provider escience_a40 proc_lsst.multi INFO: found job 17025277 in provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:201 [INFO] cancelling 17025277 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025277 on provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:146 [INFO] found job 17025278 in provider escience_a40 proc_lsst.multi INFO: found job 17025278 in provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:201 [INFO] cancelling 17025278 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025278 on provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:146 [INFO] found job 17025279 in provider escience_a40 proc_lsst.multi INFO: found job 17025279 in provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:201 [INFO] cancelling 17025279 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025279 on provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:146 [INFO] found job 17025280 in provider escience_a40 proc_lsst.multi INFO: found job 17025280 in provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:201 [INFO] cancelling 17025280 on provider escience_a40 proc_lsst.multi INFO: cancelling 17025280 on provider escience_a40 2024-03-13 23:15:05 proc_lsst.multi:146 [INFO] found job 17025256 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025256 in provider escience_rtx6k 2024-03-13 23:15:05 proc_lsst.multi:201 [INFO] cancelling 17025256 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025256 on provider escience_rtx6k 2024-03-13 23:15:05 proc_lsst.multi:146 [INFO] found job 17025257 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025257 in provider escience_rtx6k 2024-03-13 23:15:05 proc_lsst.multi:201 [INFO] cancelling 17025257 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025257 on provider escience_rtx6k 2024-03-13 23:15:05 proc_lsst.multi:146 [INFO] found job 17025258 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025258 in provider escience_rtx6k 2024-03-13 23:15:05 proc_lsst.multi:201 [INFO] cancelling 17025258 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025258 on provider escience_rtx6k 2024-03-13 23:15:06 proc_lsst.multi:146 [INFO] found job 17025259 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025259 in provider escience_rtx6k 2024-03-13 23:15:06 proc_lsst.multi:201 [INFO] cancelling 17025259 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025259 on provider escience_rtx6k 2024-03-13 23:15:06 proc_lsst.multi:146 [INFO] found job 17025260 in provider escience_rtx6k proc_lsst.multi INFO: found job 17025260 in provider escience_rtx6k 2024-03-13 23:15:06 proc_lsst.multi:201 [INFO] cancelling 17025260 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17025260 on provider escience_rtx6k 2024-03-13 23:15:06 proc_lsst.multi:146 [INFO] found job 20175 in provider local proc_lsst.multi INFO: found job 20175 in provider local 2024-03-13 23:15:06 proc_lsst.multi:201 [INFO] cancelling 20175 on provider local proc_lsst.multi INFO: cancelling 20175 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 20175 parsl.providers.local.local WARNING: Failed to kill PID: 20175 and child processes on local 2024-03-13 23:15:07 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 131.9310 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20201017_B1c_science#step0_20240313T231242Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 131.9370 seconds; current memory usage: 0.277 Gibyte, delta: 0.004 Gibyte, peak delta: 0.006 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 141.9530 seconds; current memory usage: 0.277 Gibyte, delta: 0.140 Gibyte, peak delta: 0.142 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.279 Gibyte (main), 0.279 Gibyte (largest child process) Run Id: None Run Name: DEEP_20201017_B1c_science#step0_20240313T231242Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m23.783s user 0m44.212s sys 0m22.417s